(Note: this page talks about OSM layers. There are several other uses of the word "Layer" in different contexts. In particular you can also have layers in SVG files that the Inkscape drawing program can interpret. See the description of the rules file for information about that.)
A map is created by following the instructions in each rule in the order in which the rules appear in the rule file. If the rule for a railway line comes after the rule for a motorway then railway lines will always appear to go over the top of motorways. The order of rules in the rule file caters for the most likely situation. For example, roads go over rivers, rivers go over parks.
If the default order is not correct, for example when a road goes through a tunnel under a river, the render order can be altered by tagging the road with a layer tag. By default everything is drawn at layer 0. If a way is tagged with layer=-1 it will be drawn before the main layer. If it is tagged with layer=1 it will be drawn after the main layer. The layer tag can have any integer value between -5 and 5.
If more than two ways join at the same point and they have different layer values then you will get a seam. You will not get a seam if only two ways join at a node. You may also get a rendering artifact if two ways with different layers joins at an acute angle.
For best results it is recommended that ways with different layers be kept as short as possible. For a road over a river, create a way that just spans the river. Do not tag the whole way as a different layer otherwise you will get seams all along the road wherever other roads meet it.