Hidden File Upload Gotchas That Can Break a Website

Uploading a text file or replacing a configuration file may look like a simple task, but small hidden differences in encoding, line endings, ownership, permissions, or formatting can cause major website problems.

A file can look completely normal in a text editor and still contain invisible characters that cause Apache, Nginx, PHP, Bash, WordPress, or another application to reject it.

This article outlines the most common file-upload and file-replacement gotchas, how to detect them, and how to safely deploy files across one or many websites.

1. UTF-8 BOM Characters

One of the easiest problems to miss is a UTF-8 byte order mark, usually abbreviated to BOM.

A BOM consists of three hidden bytes at the beginning of a file:

EF BB BF

Some Windows text editors add these bytes automatically when saving a file as UTF-8.

Although many applications tolerate a BOM, some configuration formats do not. An Apache .htaccess file is one example.

A normal first line may appear to be:

# BEGIN WordPress

But Apache may actually receive:

\xef\xbb\xbf# BEGIN WordPress

Apache then interprets the hidden BOM bytes as part of a directive rather than recognising the line as a comment.

The resulting error may look like this:

Invalid command '\xef\xbb\xbf#', perhaps misspelled or defined by a module not included in the server configuration

This can cause an immediate HTTP 500 Internal Server Error.

Check a file for a BOM

head -c 3 /path/to/file | od -An -tx1

head -c 3 /path/to/file | od -An -tx1

If the output begins with the following bytes, the file contains a UTF-8 BOM:

ef bb bf

Remove a BOM safely

sed '1s/^\xEF\xBB\xBF//' /path/to/original.txt > /path/to/clean.txt

sed '1s/^\xEF\xBB\xBF//' /path/to/original.txt > /path/to/clean.txt

Alternatively, remove the BOM from a file in place using Python:

python3 -c '
from pathlib import Path

path = Path("/path/to/file")
data = path.read_bytes()

if data.startswith(b"\xef\xbb\xbf"):
    path.write_bytes(data[3:])
    print("BOM removed")
else:
    print("No BOM found")
'

python3 -c '
from pathlib import Path

path = Path("/path/to/file")
data = path.read_bytes()

if data.startswith(b"\xef\xbb\xbf"):
    path.write_bytes(data[3:])
    print("BOM removed")
else:
    print("No BOM found")
'

2. Windows and Linux Line Endings

Windows and Linux use different line-ending formats.

Windows: CRLF
Linux:   LF

Many web applications handle either format without problems, but shell scripts, configuration files, cron entries, and some command-line tools may fail when Windows line endings are present.

A Bash script affected by Windows line endings may produce an error such as:

/bin/bash^M: bad interpreter: No such file or directory

Check the file format

file /path/to/file

A file with Windows line endings may be reported as:

ASCII text, with CRLF line terminators

Convert CRLF to LF

sed -i 's/\r$//' /path/to/file

sed -i 's/\r$//' /path/to/file

If dos2unix is installed, it can also be used:

dos2unix /path/to/file

3. Hidden Control Characters

Files copied from websites, word processors, AI tools, email clients, or messaging applications may contain hidden or non-standard characters.

Examples include:

non-breaking spaces;
curly quotation marks;
zero-width spaces;
smart apostrophes;
carriage returns;
Unicode dashes;
invalid control bytes.

These characters can break commands, configuration directives, CSV files, PSV files, JSON, YAML, PHP, and Bash scripts.

Display hidden characters

cat -A /path/to/file

cat -A /path/to/file

Another useful command is:

sed -n 'l' /path/to/file

sed -n 'l' /path/to/file

To inspect the raw hexadecimal bytes:

xxd /path/to/file | head -20

xxd /path/to/file | head -20

4. Incorrect File Ownership

Uploading or replacing files as the root user can leave them owned by root:root.

That may prevent the cPanel account, WordPress, PHP-FPM, or a plugin from editing or recreating the file later.

Check ownership with:

stat -c '%U:%G %a %n' /path/to/file

stat -c '%U:%G %a %n' /path/to/file

Example output:

exampleuser:exampleuser 644 /home/exampleuser/example.com/.htaccess

Correct the ownership where necessary:

chown exampleuser:exampleuser /home/exampleuser/example.com/.htaccess

chown exampleuser:exampleuser /home/exampleuser/example.com/.htaccess

5. Incorrect File Permissions

Permissions that are too restrictive may prevent the web server from reading a file. Permissions that are too open can create a security risk.

Common safe permissions are:

Regular files:       0644
Directories:         0755
Executable scripts:  0700 or 0755
Sensitive files:     0600

For a typical .htaccess file:

chmod 0644 /home/exampleuser/example.com/.htaccess

chmod 0644 /home/exampleuser/example.com/.htaccess

For a root-owned maintenance script:

chmod 700 /root/example-script.sh

chmod 700 /root/example-script.sh

6. Parent Directory Permissions

A file can have correct permissions and still be inaccessible if Apache or PHP cannot traverse one of its parent directories.

Use namei to inspect every directory in the path:

namei -l /home/exampleuser/example.com/.htaccess

namei -l /home/exampleuser/example.com/.htaccess

This displays the ownership and permissions for every directory leading to the file.

7. Invalid Configuration Directives

A configuration file may contain valid-looking directives that are not supported by the server’s current setup.

Common examples in .htaccess files include:

php_value
php_flag
Options
Header
ExpiresActive
Require
Order
Deny
Allow
SetHandler
AddHandler

A directive may work on one server or cPanel account but fail on another due to differences in:

Apache modules;
PHP-FPM or mod_php configuration;
AllowOverride settings;
PHP versions;
security modules;
virtual-host configuration.

Always validate Apache after modifying server configuration:

apachectl -t

apachectl -t

Expected output:

Syntax OK

However, remember that apachectl -t may not detect every invalid directive inside every site’s .htaccess file. Apache often reads those files only when the affected directory is requested.

8. A Valid Homepage Does Not Prove the Site Works

A website homepage can return HTTP 200 while all internal WordPress post URLs return 404.

This commonly happens when the WordPress rewrite block is missing, disabled, or ignored.

A typical WordPress rewrite section is:

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress

If .htaccess is renamed or disabled, the homepage may still work because it maps directly to the document root. Pretty permalink URLs usually depend on rewrite rules and may return 404.

Always test both the homepage and at least one internal post.

curl -skIL https://example.com/

curl -skIL https://example.com/

curl -skIL https://example.com/example-post/

curl -skIL https://example.com/example-post/

9. Reverse Proxies and CDN Caches Can Hide the Real Problem

Websites using Cloudflare, Nginx, Engintron, Varnish, or another proxy layer may return a cached response rather than the current response from Apache.

This can make troubleshooting confusing because:

a cached 500 may remain after the file is fixed;
a cached 404 may remain after rewrite rules are restored;
Cloudflare may mask the origin server headers;
Nginx and Apache may return different results.

To bypass DNS or Cloudflare and connect directly to the origin IP:

curl -skIL \
  --resolve example.com:443:203.0.113.10 \
  https://example.com/

curl -skIL \
  --resolve example.com:443:203.0.113.10 \
  https://example.com/

To test Apache directly behind Engintron:

curl -sSI \
  -H 'Host: example.com' \
  http://127.0.0.1:8080/

curl -sSI \
  -H 'Host: example.com' \
  http://127.0.0.1:8080/

To clear Engintron’s cache:

bash /opt/engintron/engintron.sh purgecache

bash /opt/engintron/engintron.sh purgecache

10. Replacing a File Is Not the Same as Replacing a Rule Block

Replacing an entire configuration file may remove unrelated rules that were previously added by:

WordPress;
Wordfence;
cPanel;
PHP handlers;
cache plugins;
redirect plugins;
security plugins;
custom server configuration.

When only one section needs changing, it is often safer to replace that specific block rather than overwrite the whole file.

For example, target content between markers such as:

# BEGIN WordPress
...
# END WordPress

This preserves unrelated custom rules elsewhere in the file.

11. Always Create a Backup

Before changing a live file, create a timestamped backup that preserves ownership, permissions, and timestamps.

cp -a /path/to/.htaccess \
  "/path/to/.htaccess.backup-$(date +%Y%m%d-%H%M%S)"

cp -a /path/to/.htaccess \
  "/path/to/.htaccess.backup-$(date +%Y%m%d-%H%M%S)"

The -a option preserves the original file metadata.

For bulk operations, keep backups in a separate root-owned directory rather than placing hundreds of backup files inside public document roots.

mkdir -p /root/file-change-backups/$(date +%Y%m%d-%H%M%S)

mkdir -p /root/file-change-backups/$(date +%Y%m%d-%H%M%S)

12. Use a Dry Run Before a Live Run

A dry run should show exactly which files would be changed without modifying anything.

A useful dry-run report should include:

WOULD CHANGE
ALREADY CORRECT
MISSING
SKIPPED
INVALID SOURCE
WRONG OWNER
WRONG PERMISSIONS

Do not assume that a successful dry run proves the replacement content is valid. A dry run normally tests file discovery and selection logic, not whether Apache or another application will accept the new content.

13. Use a Canary Deployment

Before modifying hundreds of sites, apply the change to one carefully selected test site.

After the canary change:

test the homepage;
test an internal post;
test the WordPress admin area;
check Apache and Nginx logs;
check file ownership and permissions;
confirm that no redirects or security rules were lost;
only then continue with the bulk deployment.

A one-site test can prevent a small encoding mistake from affecting hundreds of websites.

14. Validate the Source File Before Deployment

A replacement script should never blindly trust the source file.

At minimum, it should check that the source file:

exists;
is not empty;
does not contain a UTF-8 BOM;
uses the expected line endings;
does not contain unexpected control characters;
contains expected markers or directives;
passes any available syntax validation.

Example BOM preflight check:

if head -c 3 "$SOURCE_FILE" |
   od -An -tx1 |
   grep -qi 'ef bb bf'
then
    echo "ERROR: UTF-8 BOM detected in $SOURCE_FILE"
    exit 1
fi

if head -c 3 "$SOURCE_FILE" |
   od -An -tx1 |
   grep -qi 'ef bb bf'
then
    echo "ERROR: UTF-8 BOM detected in $SOURCE_FILE"
    exit 1
fi

Check that a file is not empty:

if [[ ! -s "$SOURCE_FILE" ]]; then
    echo "ERROR: Source file is missing or empty"
    exit 1
fi

if [[ ! -s "$SOURCE_FILE" ]]; then
    echo "ERROR: Source file is missing or empty"
    exit 1
fi

Check for Windows carriage returns:

if grep -q $'\r' "$SOURCE_FILE"; then
    echo "WARNING: CRLF line endings detected"
fi

if grep -q $'\r' "$SOURCE_FILE"; then
    echo "WARNING: CRLF line endings detected"
fi

15. Verify the Result After Copying

Do not assume that a successful cp command means the deployment succeeded.

Compare hashes:

sha256sum /path/to/source /path/to/destination

sha256sum /path/to/source /path/to/destination

Check ownership and permissions:

stat -c '%U:%G %a %n' /path/to/destination

stat -c '%U:%G %a %n' /path/to/destination

Check the first few bytes:

head -c 8 /path/to/destination | od -An -tx1

head -c 8 /path/to/destination | od -An -tx1

Then make an actual HTTP request:

curl -skIL https://example.com/

curl -skIL https://example.com/

curl -skIL https://example.com/example-post/

curl -skIL https://example.com/example-post/

16. Check the Error Logs Immediately

Error logs often identify the exact problem faster than repeatedly changing files.

Apache error log:

tail -n 100 /usr/local/apache/logs/error_log

tail -n 100 /usr/local/apache/logs/error_log

Nginx error log:

tail -n 100 /var/log/nginx/error.log

tail -n 100 /var/log/nginx/error.log

Follow the Apache log while reproducing the problem:

tail -f /usr/local/apache/logs/error_log

tail -f /usr/local/apache/logs/error_log

An error such as the following immediately reveals a BOM problem:

Invalid command '\xef\xbb\xbf#'

A Safer Bulk File-Replacement Workflow

The following workflow should be used whenever files are replaced or altered across multiple websites.

Back up the source and destination files.
Confirm the source file is not empty.
Check for UTF-8 BOM bytes.
Check line endings.
Inspect for hidden control characters.
Validate syntax where possible.
Confirm expected ownership and permissions.
Run a dry run.
Deploy to one canary site.
Test the homepage and an internal URL.
Inspect Apache, Nginx, PHP, and application logs.
Deploy to a small second batch.
Only then deploy to all remaining sites.
Run an automated post-deployment HTTP check.
Keep a rollback path available until verification is complete.

Example Preflight Script

The following Bash example checks a source file before it is used in a bulk replacement operation.

#!/usr/bin/env bash

set -euo pipefail

SOURCE_FILE="${1:-}"

if [[ -z "$SOURCE_FILE" ]]; then
    echo "Usage: $0 /path/to/source-file"
    exit 1
fi

if [[ ! -f "$SOURCE_FILE" ]]; then
    echo "ERROR: File not found: $SOURCE_FILE"
    exit 1
fi

if [[ ! -s "$SOURCE_FILE" ]]; then
    echo "ERROR: File is empty: $SOURCE_FILE"
    exit 1
fi

echo "Checking: $SOURCE_FILE"

if head -c 3 "$SOURCE_FILE" |
   od -An -tx1 |
   grep -qi 'ef bb bf'
then
    echo "ERROR: UTF-8 BOM detected"
    exit 1
fi

if grep -q $'\r' "$SOURCE_FILE"; then
    echo "WARNING: Windows CRLF line endings detected"
else
    echo "Line endings: OK"
fi

echo
echo "File type:"
file "$SOURCE_FILE"

echo
echo "Ownership and permissions:"
stat -c '%U:%G %a %n' "$SOURCE_FILE"

echo
echo "First five lines:"
sed -n '1,5p' "$SOURCE_FILE"

echo
echo "First 16 bytes:"
head -c 16 "$SOURCE_FILE" | od -An -tx1

echo
echo "Preflight checks completed"

#!/usr/bin/env bash

set -euo pipefail

SOURCE_FILE="${1:-}"

if [[ -z "$SOURCE_FILE" ]]; then
    echo "Usage: $0 /path/to/source-file"
    exit 1
fi

if [[ ! -f "$SOURCE_FILE" ]]; then
    echo "ERROR: File not found: $SOURCE_FILE"
    exit 1
fi

if [[ ! -s "$SOURCE_FILE" ]]; then
    echo "ERROR: File is empty: $SOURCE_FILE"
    exit 1
fi

echo "Checking: $SOURCE_FILE"

if head -c 3 "$SOURCE_FILE" |
   od -An -tx1 |
   grep -qi 'ef bb bf'
then
    echo "ERROR: UTF-8 BOM detected"
    exit 1
fi

if grep -q $'\r' "$SOURCE_FILE"; then
    echo "WARNING: Windows CRLF line endings detected"
else
    echo "Line endings: OK"
fi

echo
echo "File type:"
file "$SOURCE_FILE"

echo
echo "Ownership and permissions:"
stat -c '%U:%G %a %n' "$SOURCE_FILE"

echo
echo "First five lines:"
sed -n '1,5p' "$SOURCE_FILE"

echo
echo "First 16 bytes:"
head -c 16 "$SOURCE_FILE" | od -An -tx1

echo
echo "Preflight checks completed"

Final Takeaway

The most dangerous file-upload problems are often invisible.

A file can have the correct name, readable content, correct ownership, and correct permissions while still containing a three-byte encoding marker that takes hundreds of websites offline.

The safest approach is to treat every bulk file replacement as a deployment rather than a simple copy operation.

Always check the source, back up the destination, test one site, inspect the logs, verify the result, and only then continue across the full server.

M	T	W	T	F	S	S
		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30	31

500 Internal Server Error – UTF-8 BOM Characters

Hidden File Upload Gotchas That Can Break a Website

1. UTF-8 BOM Characters

Check a file for a BOM

Remove a BOM safely

2. Windows and Linux Line Endings

Check the file format

Convert CRLF to LF

3. Hidden Control Characters

Display hidden characters

4. Incorrect File Ownership

5. Incorrect File Permissions

6. Parent Directory Permissions

7. Invalid Configuration Directives

8. A Valid Homepage Does Not Prove the Site Works

9. Reverse Proxies and CDN Caches Can Hide the Real Problem

10. Replacing a File Is Not the Same as Replacing a Rule Block

11. Always Create a Backup

12. Use a Dry Run Before a Live Run

13. Use a Canary Deployment

14. Validate the Source File Before Deployment

15. Verify the Result After Copying

16. Check the Error Logs Immediately

A Safer Bulk File-Replacement Workflow

Example Preflight Script

Final Takeaway

Contact Form 7 Not Sending Email

Script to replace WordPress Core Files on a Malware Infected site

WordPress Malware Remediation Incident – Case Study

WordPress Server Malware Case Study: Detection and Cleanup

Leave a Reply Cancel reply

Hidden File Upload Gotchas That Can Break a Website

1. UTF-8 BOM Characters

Check a file for a BOM

Remove a BOM safely

2. Windows and Linux Line Endings

Check the file format

Convert CRLF to LF

3. Hidden Control Characters

Display hidden characters

4. Incorrect File Ownership

5. Incorrect File Permissions

6. Parent Directory Permissions

7. Invalid Configuration Directives

8. A Valid Homepage Does Not Prove the Site Works

9. Reverse Proxies and CDN Caches Can Hide the Real Problem

10. Replacing a File Is Not the Same as Replacing a Rule Block

11. Always Create a Backup

12. Use a Dry Run Before a Live Run

13. Use a Canary Deployment

14. Validate the Source File Before Deployment

15. Verify the Result After Copying

16. Check the Error Logs Immediately

A Safer Bulk File-Replacement Workflow

Example Preflight Script

Final Takeaway

Similar Posts

Leave a Reply Cancel reply