What can a hacker do with an ip address?

To answer your question of what can a hacker do with an IP address - I think it may be better to illustrate how hacking actually works to put your mind at ease. Keep in mind for this I'm talking about WAN (internet) IP's.

Firstly as has been pointed out there are IP addresses, which is simply a way to look at your router. Everyone on the internet has one, and you can imagine that if simply knowing an IP address allowed us to hack people we'd have pretty big problems! Secondly an important thing to know is each IP address has ports associated with it. Think of them as a door into a building. They number from 1 - 65535, and if you are running a program that faces to the internet, typically the relevant port number will be open, allowing people to 'come in' and interact with the program. (Relevant common ports: http://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers). Finally we have programs which run behind these doors. It may be a FTP server, mail server, or any of the programs that interact with the internet. Think of these like self contained rooms, even if someone gets to your address, goes in the door, they can only play in the little self contained room, they don't have access to all your house.

So we have IP addresses, ports and programs. Think of them as a physical address, doorway into a room, and a room itself respectively. Now the vast vast majority of non-commercial connections will not have any programs running that are facing the internet. Combined with the fact that almost every non-commercial connection out there will be using a thing called NAT, it means essentially you don't have any doors in your address facing the public, and hence you're for all relevant purposes you're completely safe. This is what people talk about when they say firewalls, essentially something that blocks the ports from the public. Let's say however that you enjoy being able to send files to your home computer so you have an FTP server running on your PC, with port 21 open to the internet so you can connect to it.

Using this involves you finding your IP address (physical address), going to the relevant door (port 21), and then using the program (mucking around in the room). Most of the time this room will be completely safe for people to play around in, they can't go anywhere else. Sometimes though it may have a vulnerability in it, which you can think of as a weak wall somewhere. What a hacker does is look for an IP address, see that a port is open and find out what program is running behind it. They then try and find a vulnerability for it, and if they exploit that they can essentially break down the wall and wreak havoc in your house. That, in a very over simplified sense, is how hacking works.

So - let's look at all the things you need in combination in order to be worried.

IP Address Open ports (Because of the prevalence of NAT, very few home connections have this) Vulnerable programs running (This is why it's important to patch if you do have things running, as security flaws are found patches are released to fix them up) You are most likely only satisfying 1/3, as are the vast majority of other internet users.