Sometimes there might be a problem with the server on which your website is hosted on, making it difficult to access. That amount of time that a given website is inaccessible is what we refer to as downtime. In case engine spiders happen to visit your website during such periods of website inaccessibility, they will record that your site is inaccessible and move to the next site. If they visit your site for a number of times and find the same thing, your working is likely to be ranked lower and seem unreliable. Your site being displayed as unreliable reflects that to your search engine therefore reflecting a bad image on them.
The speed of loading a page is one of the factors of search engine algorithms. A loading speed slower than average does not necessarily mean the fault is with your web host. The problem might be in your coding and configuration, but if you are certain that there is optimization in your site, then it is recommended you request your hosting company to shift your site to another server. Your page is most likely to load a bit fast if your site is moved to a private server or to a less busy one as compared to when it is hosted by a sever that shares resources with a lot of sites.
Even though security will have an indirect effect on your SEO, it will most probably imply negative effects in case you get hacked.
The above factors will make a great different when it comes to how your site will get ranked, therefore determining reliability of your site.