If you have an installation of Concrete5 you may notice that navigating to example.com/config will give you a list of files available in that folder. This type of access is different than the robots.txt file which communicates to search engine crawlers what to allow and disallow.

Setting up an .htaccess file will block access to these directories from a browser but you can also have the plain text on white server response 'Forbidden' and 'Not Found' pages redirect to the standard concrete5 'page not found', and if you have set up your config/site_theme_paths.php file correctly they will redirect to the 'page not found' styled in your theme.

1) Create or modify your .htaccess file in the root directory of your concrete5 install.

NOTE* If the .htaccess file is not there you can create one. Sometimes depending on your FTP connection or FTP software you cannot see certain file types and must enable viewing hidden or cloaked file types. I have used FileZilla to upload files to and from my server and it works great for viewing the .htaccess and other hidden filetypes.

2) Inside your .htaccess file place this text and save.

Options -Indexes
DirectoryIndex index.php
ErrorDocument 401 /index.php
ErrorDocument 400 /index.php
ErrorDocument 403 /index.php
ErrorDocument 404 /index.php
ErrorDocument 500 /index.php

3) Now go check your site by navigating to example.com/config, you should be redirected to either the concrete 'page not found' or your custom themes 'page not found' page.

4) Setting custom 404 and 'page not found' Register, Login and error pages is easy with concrete5. Find your site_theme_paths.php file in the config folder of your concrete5 install and make sure this text is in there.

$v = View::getInstance();

$v->setThemeByPath('/login', "custom_theme_name");
$v->setThemeByPath('/register', "custom_theme_name");
$v->setThemeByPath('/page_forbidden', "custom_theme_name");
$v->setThemeByPath('/page_not_found', "custom_theme_name");
$v->setThemeByPath('/user_error', "custom_theme_name");

5) Setting up your robots.txt file in concrete5 on a basic install this may already be done for you, if not create the file and place this text inside & save.

User-agent: *
Disallow: /blocks 
Disallow: /concrete 
Disallow: /config 
Disallow: /controllers 
Disallow: /css 
Disallow: /elements 
Disallow: /helpers 
Disallow: /jobs 
Disallow: /js 
Disallow: /languages 
Disallow: /libraries 
Disallow: /mail 
Disallow: /models 
Disallow: /packages 
Disallow: /single_pages 
Disallow: /themes 
Disallow: /tools
Disallow: /updates

Now when search engine crawlers hit your site they won't look inside of disallowed folders

Some extra explanation 1) .htaccess

Options -Indexes

This blocks access to showing directory lists

DirectoryIndex index.php

This sets the index.php and the default page. This is naturally the case but if you had a page than was anything-non-standard.php you could set this here. this also works for other types of web documents like .html etc...

ErrorDocument 401 /index.php
ErrorDocument 400 /index.php
ErrorDocument 403 /index.php
ErrorDocument 404 /index.php
ErrorDocument 500 /index.php

These all set the server error pages to point to the index.php file. I haven't done it but I am sure you could be more specific in the url string if you wanted to create separate 401, 400, 403, 404, & 500 error pages.

2) Site theme paths. The basic idea here is that you are setting the theme as a part of the path which enables custom pages to load, this is different than customizing the content on these pages.

As a side note the 'custom_theme_name' is literally the name of the folder your theme contents are in, if it is customthemename or CustomThemeName it should work but dashes - do not work. Example custom-theme-name will not properly to redirect the single pages.

3)robots.txt

User-agent: *

This asterisk here is specifying all user agent types

Disallow: /blocks 
Disallow: /concrete 
Disallow: /config 
Disallow: /controllers 
Disallow: /css 
Disallow: /elements 
Disallow: /helpers 
Disallow: /jobs 
Disallow: /js 
Disallow: /languages 
Disallow: /libraries 
Disallow: /mail 
Disallow: /models 
Disallow: /packages 
Disallow: /single_pages 
Disallow: /themes 
Disallow: /tools
Disallow: /updates

Here you can allow & disallow directory indexing from search engines, you can be vague and only set the top level directory which blocks the whole directory from being indexed or you can be more complex & allow top level access but block specific internal areas,

Disallow:/example/noindex

Enjoy!

EDIT: for more information on .htaccess & further apache things visit the source! •_• http://httpd.apache.org/docs/current/custom-error.html