What does the fin scan attack do?

It's an older attack originally intended to be a "sneaky, firewall bypass" that was dependent on a few factors that are now uncommon today: old Unix OSes, lack of stateful firewalls, lack of NIDS/NIPS, etc. It can still be useful when testing (i.e., as a fingerprinting technique not an attack per se) completely new or novel TCP/IP stacks (or just new to you or your environment), which is rare but can happen.

Here is a modern replacement, the TCP protocol FIN scan:

nmap --reason -n -Pn --packet-trace -g 80 -sO -p 6 

Which is almost exactly the same as the TCP ACK scan (which can be used to map hosts, open ports, firewall rulesets, etc with the caveat that some NIPS, IDS, and modern firewalls will detect -- with another situation-specific event where perhaps it will not notify incident responders or Security Operations Centres because they have more important things to look at these days):

  nmap --reason -n -Pn --packet-trace -g 80 -sA -p 80

But the outputs are slightly different and you can see the other packet-level differences as well. What you are looking for in order to develop a more advanced technique is to identify the subtleties in the RST packets and their window sizes. If you get non-zero window sizes, then you may want to switch to using the TCP Window scan instead of the TCP ACK scan. For more information, see http://nmap.org/book/man-port-scanning-techniques.html Some other techniques are found in the NSE guide, such as the firewalk and firewall-bypass scripts. However, there are many other techniques including BNAT, fragroute, osstmm-afd, 0trace, lft, and potentially others that detect other inline, non-firewall devices such as WAFs, IDS, IPS, reverse proxies, gateways, and deception systems such as honeypots or active defences. You will want to be aware of all of this and more if you are performing a network penetration test, but they come in handy for troubleshooting all sorts of network and security issues.