Composition in photography is no more or less than finding a pleasing lay-out for the subject(s) in your frame that works best. This means that you are looking for the right distribution of elements throughout your frame, but what is right and what is wrong? There are many composition rules that might be helpful, but in the end it really comes down to you telling the story of the scene you are capturing in a way that is uniquely yours.
Believe it or not, but your way of looking at a scene will never be entirely the same as mine and this is exactly as it should be. Your way of looking has been shaped over your entire life and has its roots in experiences, feelings, passions and interests. The brain simply has a wonderful way of overlooking stuff it does not want to see.
Most people see the world horizontally and our cameras are designed in that same way, but does this mean that you always need to shoot horizontally? In portrait photography it is of course widely accepted to use a vertical lay-out, but you don't see it as much in landscape photography. This is because most of us will look at a scene in a horizontal way (I actually don't, which seems to be quite rare).
In these days where we look at everything on our mobile phones, more and more vertical (4:5) pictures are popping up on Instagram and Facebook. This is because horizontal pictures take up less of the screen on Instagram and have a lower impact because of it. A vertical picture (and Instagram allows only the 4:5 ratio) takes up more of the screen and this has become the reason for many photographers to shoot either vertically for social media or to shoot horizontally and crop the picture into a vertical later.
I would suggest to approach the lay-out issue from an artistic point of view. The good thing about being an artist is that you can make all the decisions and if you make a conscious decision about the lay-out of your pictures, chances are that your work will improve.
So how should you pick the "right" lay-out? Most of the time I ask students how they perceive a scene. As you know, I mostly work in forests in which case you should ask yourself questions like:
~ What comes to mind when I see this scene? Does the scene feel enclosed or do I want it to feel that way? Is this a narrow path and do I want to emphasise that? Is the height of the trees the thing that made me grab my camera?
In all of these cases it might be advisable to opt for a vertical lay-out
~ Do I want to make this scene feel intimate or overpowering? Most of the time I opt for horizontal for intimate scenes with lower overhanging branches and vertical for overpowering scenes. Is this path looking wide or narrow? Is the essence of this scene vertically directed or horizontally?
Let me explain the last question. There are scenes that are most definitely vertically oriented like a tall tree, or perhaps a small waterfall falling from high above, subjects that are themselves vertical in shape etc
And then there are scenes that are definitely horizontally oriented like a scene in which someone is moving from left to right in the frame, you would usually allow room for this person to move in. Wide vistas are of course naturally horizontal in appearance as well and scenes in which you want to emphasise the the sense of space.
But... a lot of scenes can be perceived as horizontal and vertical and in this case I usually take pictures in both orientations. I love vertical images as they are more like I see the world, they can be more dynamic and you can include more of the scene from foreground to background in the frame.
Here is an example of one waterfall and two lay-outs in which case one tells a way better story of this waterfall than the other. This is a waterfall that does not come from high above, the direction of the waterfall is not vertical and so the horizontal one just tells the story of this waterfall much better than the vertical picture which tells the story of high beech trees with a bit of water running below.
In the following example both orientations work equally well, but one conveys a sense of height of these old trees and the other one has the emphasis more on the tree trunks
And another example of a scene of which I like the horizontal picture just as much as the vertical one
If you need a guideline it is often best to use horizontal lay-outs to convey a sense of space, for obviously horizontally orientated subjects and to allow movement in the frame from left to right or right to left. It is also great for emphasising the width of things and in a forest it places emphasis on the trunks rather than the height of the trees. It is best to opt for a vertical lay-out to convey height, power, for vertical objects and to emphasise the narrow or enclosed nature of a scene and if there is more vertical movement (think of a waterfall for example). If the direction of your eye goes from top to bottom and back again, instead of from side to side...you need to opt for a vertical lay-out. I will explain this with the example of a bird of prey looking straight down at let's say a frog and it is grey cloudless day. The direction in which the bird looks is from the top of the frame to the bottom and this forces us to look in that same direction. If there is little else of interest in this scene (especially on this grey day), it might be wise to go for a vertical frame to emphasise the story of this bird looking down at this frog.
I highly recommend getting an L-bracket on your camera for vertical pictures, because it is very tedious to swivel the ball of your tripod into the vertical position and most of the time it destabilises the tripod. An L-bracket does not have to be very expensive and it will save your from so much frustration. They are camera specific, so you need to buy one that fits your particular camera model and make sure that you can still access your battery and card compartment.
One last tip....I use the vertical format to shoot panoramas as well. Your camera needs to be exactly level though for this to work well and I sometimes go up to 7 frames that I stitch together in Lightroom or Photoshop, which is sometimes the only way I can capture a scene exactly like I have in mind. remember though that the file size can be huge, depending on the resolution of your camera and it can take a long time for your computer to process a panorama shot like this.
If you want to learn all about forest photography, I have an eBook called : The Magic Of Forest Photography that goes deeply into how to capture enchanting pictures, what conditions work well and also how I process my forest pictures.