I agree, it's fun just to fly the ship, that's definitely a good thing, since you don't want a game where the most basic task is boring.
Now, add weaponry, enemy ships, nebulae, asteroid fields, planets you can fly into (and orbit), gravity, a ramscoop, a cloaking device, fuel, multiple ships, and hyperspace travel (put a speed cap for crying out loud!), I'll be hooked.
Add communications, quests, a planet lander ability (a la Starflight), mining, trade, upgrades and evil villains, and you'll have yourself a game.
Oh, yeah, time dilation and realistic acceleration. Firstly, the faster any mass travels relative to another mass, the slower the atoms of the first mass will rotate relative to the second mass, therefore making movement slower, or effectively, slowing time. So if you have mass1 (your starship) and mass2 (a star), and you set the star as the control mass, with a relative velocity of 0, then the faster your starship approaches that star, the slower time will pass relative to the star. (And, actually, relative to most of the stars in a galaxy, since they all have the same velocity as they circle the galactic core, more or less)
This means 4 years could pass in "real" time (there's no such thing, but for "real" say "galactic standard" or something), and only 2 days on board the ship, due to the difference in relative velocity if mass1 is traveling at something like 99.994% the speed of light relatively.
Just something to consider if you decide not to cap sub-light travel. (Hyperspace and other such pseudo-science devices completely ignore this)
The second thing is acceleration. In your game, you can keep accelerating at a constant rate. In reality, this couldn't happen. Again, going by the standard of light, if you identify one velocity as being A, the speed of light being C, and a velocity directly in between a and c as being b like such:
a ------------------------------------ b ------------------------------------ c
..................x..................................................4x
Let's say a = 0. Therefore, b = 0.5c. In a vacuum (like space), the amount of effort to go from acceleration b to acceleration c is FOUR TIMES the amount required to go from a to b. In fact, this is applicable no matter what values you use. The amount of effort to go from 99.994c to 99.995c is 4x going from 99.993% c to 99.994% c. This happens because as you gain velocity, you gain mass. Actually, it'd be impossible to reach 1c because you'd obtain infinite mass, and therefore it'd require infinite energy.
All that said, sub light acceleration should either slow down the faster you travel, or it should require more fuel. =P (This is just a game mechanics thing, actually... going insanely fast and not paying any penalty for it is just... off)