If you have an installation of Concrete5 you may notice that navigating to example.com/config will give you a list of files available in that folder. This type of access is different than the robots.txt file which communicates to search engine crawlers what to allow and disallow.
Setting up an .htaccess file will block access to these directories from a browser but you can also have the plain text on white server response 'Forbidden' and 'Not Found' pages redirect to the standard concrete5 'page not found', and if you have set up your config/site_theme_paths.php file correctly they will redirect to the 'page not found' styled in your theme.
1) Create or modify your .htaccess file in the root directory of your concrete5 install.
NOTE* If the .htaccess file is not there you can create one. Sometimes depending on your FTP connection or FTP software you cannot see certain file types and must enable viewing hidden or cloaked file types. I have used FileZilla to upload files to and from my server and it works great for viewing the .htaccess and other hidden filetypes.
2) Inside your .htaccess file place this text and save.
Options -Indexes DirectoryIndex index.php ErrorDocument 401 /index.php ErrorDocument 400 /index.php ErrorDocument 403 /index.php ErrorDocument 404 /index.php ErrorDocument 500 /index.php
3) Now go check your site by navigating to example.com/config, you should be redirected to either the concrete 'page not found' or your custom themes 'page not found' page.
4) Setting custom 404 and 'page not found' Register, Login and error pages is easy with concrete5. Find your site_theme_paths.php file in the config folder of your concrete5 install and make sure this text is in there.
$v = View::getInstance(); $v->setThemeByPath('/login', "custom_theme_name"); $v->setThemeByPath('/register', "custom_theme_name"); $v->setThemeByPath('/page_forbidden', "custom_theme_name"); $v->setThemeByPath('/page_not_found', "custom_theme_name"); $v->setThemeByPath('/user_error', "custom_theme_name");
5) Setting up your robots.txt file in concrete5 on a basic install this may already be done for you, if not create the file and place this text inside & save.
User-agent: * Disallow: /blocks Disallow: /concrete Disallow: /config Disallow: /controllers Disallow: /css Disallow: /elements Disallow: /helpers Disallow: /jobs Disallow: /js Disallow: /languages Disallow: /libraries Disallow: /mail Disallow: /models Disallow: /packages Disallow: /single_pages Disallow: /themes Disallow: /tools Disallow: /updates
Now when search engine crawlers hit your site they won't look inside of disallowed folders
Some extra explanation 1) .htaccess
This blocks access to showing directory lists
This sets the index.php and the default page. This is naturally the case but if you had a page than was anything-non-standard.php you could set this here. this also works for other types of web documents like .html etc...
ErrorDocument 401 /index.php ErrorDocument 400 /index.php ErrorDocument 403 /index.php ErrorDocument 404 /index.php ErrorDocument 500 /index.php
These all set the server error pages to point to the index.php file. I haven't done it but I am sure you could be more specific in the url string if you wanted to create separate 401, 400, 403, 404, & 500 error pages.
2) Site theme paths. The basic idea here is that you are setting the theme as a part of the path which enables custom pages to load, this is different than customizing the content on these pages.
As a side note the 'custom_theme_name' is literally the name of the folder your theme contents are in, if it is customthemename or CustomThemeName it should work but dashes - do not work. Example custom-theme-name will not properly to redirect the single pages.
This asterisk here is specifying all user agent types
Disallow: /blocks Disallow: /concrete Disallow: /config Disallow: /controllers Disallow: /css Disallow: /elements Disallow: /helpers Disallow: /jobs Disallow: /js Disallow: /languages Disallow: /libraries Disallow: /mail Disallow: /models Disallow: /packages Disallow: /single_pages Disallow: /themes Disallow: /tools Disallow: /updates
Here you can allow & disallow directory indexing from search engines, you can be vague and only set the top level directory which blocks the whole directory from being indexed or you can be more complex & allow top level access but block specific internal areas,
EDIT: for more information on .htaccess & further apache things visit the source! •_• http://httpd.apache.org/docs/current/custom-error.html