Removing Special Characters from Unicode Strings in C#: Complete Guide

6 minute read

Working with Unicode strings and removing special characters is a common requirement in text processing applications. Whether you’re sanitizing user input, preparing data for databases, or cleaning text for analysis, understanding different approaches to handle special characters in Unicode strings is essential.

Understanding Unicode and Special Characters

Unicode is a standard that provides a unique number for every character, regardless of platform, program, or language. When dealing with Unicode text, special characters can include:

Punctuation marks: !, @, #, $, %, ^, &, *
Mathematical symbols: +, -, =, <, >
Currency symbols: $, €, ¥, ₹
Diacritical marks: é, ñ, ü, ç
Whitespace characters: tabs, line breaks, non-breaking spaces
Control characters and formatting marks

Method 1: Simple Character Replacement

The most basic approach removes specific known characters:

using System;

class SimpleCharacterRemoval
{
    public static string RemoveSpecificCharacters(string input)
    {
        if (string.IsNullOrEmpty(input))
            return input;

        return input.Replace("@", "")
                   .Replace("#", "")
                   .Replace("$", "")
                   .Replace("%", "")
                   .Replace("^", "")
                   .Replace("&", "")
                   .Replace("*", "")
                   .Replace("|", "")
                   .Replace("_", "");
    }

    public static void TestSimpleRemoval()
    {
        string input = "SG@%@sgs th? g#%@^@#$ chào^#^$#!abc35| _ sgs _35 hello world không gsg";
        string output = RemoveSpecificCharacters(input);

        Console.WriteLine("=== Simple Character Replacement ===");
        Console.WriteLine($"Input:  {input}");
        Console.WriteLine($"Output: {output}");
    }
}

Pros: Simple and fast for known characters
Cons: Limited flexibility, requires modification for new characters

Method 2: Regular Expression Approach

Regular expressions provide powerful pattern matching for character removal:

using System;
using System.Text.RegularExpressions;

class RegexCharacterRemoval
{
    // Remove all non-alphanumeric characters (keeping only letters and numbers)
    public static string KeepAlphanumericOnly(string input)
    {
        if (string.IsNullOrEmpty(input))
            return input;

        return Regex.Replace(input, @"[^a-zA-Z0-9\s]", "");
    }

    // Remove specific special characters using character class
    public static string RemoveSpecialCharacters(string input)
    {
        if (string.IsNullOrEmpty(input))
            return input;

        return Regex.Replace(input, @"[@#$%^&*|_]", "");
    }

    // Keep only Unicode letters, numbers, and whitespace
    public static string KeepUnicodeLettersAndNumbers(string input)
    {
        if (string.IsNullOrEmpty(input))
            return input;

        return Regex.Replace(input, @"[^\p{L}\p{N}\s]", "");
    }

    // Remove punctuation but keep Unicode letters and numbers
    public static string RemovePunctuation(string input)
    {
        if (string.IsNullOrEmpty(input))
            return input;

        return Regex.Replace(input, @"\p{P}", "");
    }

    // Custom pattern: Remove everything except letters, numbers, spaces, and specific characters
    public static string CustomFilter(string input, string allowedSpecialChars = "")
    {
        if (string.IsNullOrEmpty(input))
            return input;

        string pattern = $@"[^\p\p\s{Regex.Escape(allowedSpecialChars)}]";
        return Regex.Replace(input, pattern, "");
    }

    public static void TestRegexMethods()
    {
        string input = "SG@%@sgs thể g#%@^@#$ chào^#^$#!abc35| _ sgs _35 hello world không gsg";

        Console.WriteLine("=== Regular Expression Methods ===");
        Console.WriteLine($"Original: {input}\n");

        Console.WriteLine($"Alphanumeric only: {KeepAlphanumericOnly(input)}");
        Console.WriteLine($"Remove [@#$%^&*|_]: {RemoveSpecialCharacters(input)}");
        Console.WriteLine($"Unicode letters/numbers: {KeepUnicodeLettersAndNumbers(input)}");
        Console.WriteLine($"Remove punctuation: {RemovePunctuation(input)}");
        Console.WriteLine($"Custom (allow .-): {CustomFilter(input, ".-")}");
    }
}

Important Regex Patterns:

Pattern	Description
`[^a-zA-Z0-9\s]`	Remove everything except ASCII letters, numbers, and spaces
`\p{P}`	Matches any punctuation character
`\p{L}`	Matches any Unicode letter
`\p{N}`	Matches any Unicode number
`\p{S}`	Matches any Unicode symbol
`[^\p{L}\p{N}\s]`	Keep only Unicode letters, numbers, and whitespace

Method 3: LINQ-Based Approach

Using LINQ for functional-style character filtering:

using System;
using System.Linq;
using System.Text;

class LinqCharacterRemoval
{
    public static string RemoveCharactersLinq(string input, Func<char, bool> predicate)
    {
        if (string.IsNullOrEmpty(input))
            return input;

        return new string(input.Where(predicate).ToArray());
    }

    public static string KeepAlphanumericLinq(string input)
    {
        return RemoveCharactersLinq(input, c => char.IsLetterOrDigit(c) || char.IsWhiteSpace(c));
    }

    public static string RemoveSpecialCharactersLinq(string input)
    {
        char[] specialChars = { '@', '#', '$', '%', '^', '&', '*', '|', '_' };
        return RemoveCharactersLinq(input, c => !specialChars.Contains(c));
    }

    public static string KeepUnicodeLettersLinq(string input)
    {
        return RemoveCharactersLinq(input, c => char.IsLetter(c) || char.IsDigit(c) || char.IsWhiteSpace(c));
    }

    public static void TestLinqMethods()
    {
        string input = "SG@%@sgs thể g#%@^@#$ chào^#^$#!abc35| _ sgs _35 hello world không gsg";

        Console.WriteLine("=== LINQ-Based Methods ===");
        Console.WriteLine($"Original: {input}\n");

        Console.WriteLine($"Alphanumeric (LINQ): {KeepAlphanumericLinq(input)}");
        Console.WriteLine($"Remove specials (LINQ): {RemoveSpecialCharactersLinq(input)}");
        Console.WriteLine($"Unicode letters (LINQ): {KeepUnicodeLettersLinq(input)}");
    }
}

Method 4: StringBuilder for Performance

For large strings or frequent operations, StringBuilder provides better performance:

using System;
using System.Text;

class StringBuilderCharacterRemoval
{
    public static string RemoveCharactersStringBuilder(string input, char[] charsToRemove)
    {
        if (string.IsNullOrEmpty(input))
            return input;

        var sb = new StringBuilder(input.Length);
        
        foreach (char c in input)
        {
            if (!Array.Exists(charsToRemove, ch => ch == c))
            {
                sb.Append(c);
            }
        }

        return sb.ToString();
    }

    public static string KeepOnlyLettersAndNumbers(string input)
    {
        if (string.IsNullOrEmpty(input))
            return input;

        var sb = new StringBuilder(input.Length);
        
        foreach (char c in input)
        {
            if (char.IsLetterOrDigit(c) || char.IsWhiteSpace(c))
            {
                sb.Append(c);
            }
        }

        return sb.ToString();
    }

    public static void TestStringBuilderMethods()
    {
        string input = "SG@%@sgs thể g#%@^@#$ chào^#^$#!abc35| _ sgs _35 hello world không gsg";
        char[] specialChars = { '@', '#', '$', '%', '^', '&', '*', '|', '_' };

        Console.WriteLine("=== StringBuilder Methods ===");
        Console.WriteLine($"Original: {input}\n");

        Console.WriteLine($"Remove array of chars: {RemoveCharactersStringBuilder(input, specialChars)}");
        Console.WriteLine($"Letters and numbers only: {KeepOnlyLettersAndNumbers(input)}");
    }
}

Method 5: Advanced Unicode Handling

Comprehensive Unicode character classification and handling:

using System;
using System.Globalization;
using System.Text;
using System.Text.RegularExpressions;

class AdvancedUnicodeHandling
{
    public static string NormalizeAndClean(string input)
    {
        if (string.IsNullOrEmpty(input))
            return input;

        // Normalize Unicode (decompose accented characters)
        string normalized = input.Normalize(NormalizationForm.FormD);
        
        var sb = new StringBuilder();
        
        foreach (char c in normalized)
        {
            UnicodeCategory category = CharUnicodeInfo.GetUnicodeCategory(c);
            
            // Keep letters, digits, and spaces; skip combining diacritical marks
            if (category != UnicodeCategory.NonSpacingMark)
            {
                if (char.IsLetterOrDigit(c) || char.IsWhiteSpace(c))
                {
                    sb.Append(c);
                }
            }
        }

        return sb.ToString().Normalize(NormalizationForm.FormC);
    }

    public static string RemoveByUnicodeCategory(string input, params UnicodeCategory[] categoriesToRemove)
    {
        if (string.IsNullOrEmpty(input))
            return input;

        var sb = new StringBuilder();
        
        foreach (char c in input)
        {
            UnicodeCategory category = CharUnicodeInfo.GetUnicodeCategory(c);
            
            if (!Array.Exists(categoriesToRemove, cat => cat == category))
            {
                sb.Append(c);
            }
        }

        return sb.ToString();
    }

    public static void TestAdvancedMethods()
    {
        string input = "Héllo Wörld! Café résumé naïve 测试 العربية русский";

        Console.WriteLine("=== Advanced Unicode Handling ===");
        Console.WriteLine($"Original: {input}\n");

        Console.WriteLine($"Normalized and cleaned: {NormalizeAndClean(input)}");
        
        // Remove punctuation and symbols
        string withoutPunctuation = RemoveByUnicodeCategory(input, 
            UnicodeCategory.OtherPunctuation, 
            UnicodeCategory.OpenPunctuation, 
            UnicodeCategory.ClosePunctuation,
            UnicodeCategory.ConnectorPunctuation,
            UnicodeCategory.DashPunctuation,
            UnicodeCategory.FinalQuotePunctuation,
            UnicodeCategory.InitialQuotePunctuation,
            UnicodeCategory.OtherPunctuation);
            
        Console.WriteLine($"Without punctuation: {withoutPunctuation}");
    }
}

Performance Comparison

Here’s a benchmark comparison of different methods:

using System;
using System.Diagnostics;

class PerformanceComparison
{
    public static void BenchmarkMethods()
    {
        string input = "SG@%@sgs thể g#%@^@#$ chào^#^$#!abc35| _ sgs _35 hello world không gsg";
        int iterations = 100000;

        Console.WriteLine("=== Performance Comparison ===");
        Console.WriteLine($"Testing with {iterations:N0} iterations\n");

        // Test String.Replace method
        var sw = Stopwatch.StartNew();
        for (int i = 0; i < iterations; i++)
        {
            SimpleCharacterRemoval.RemoveSpecificCharacters(input);
        }
        sw.Stop();
        Console.WriteLine($"String.Replace: {sw.ElapsedMilliseconds} ms");

        // Test Regex method
        sw.Restart();
        for (int i = 0; i < iterations; i++)
        {
            RegexCharacterRemoval.RemoveSpecialCharacters(input);
        }
        sw.Stop();
        Console.WriteLine($"Regex: {sw.ElapsedMilliseconds} ms");

        // Test LINQ method
        sw.Restart();
        for (int i = 0; i < iterations; i++)
        {
            LinqCharacterRemoval.RemoveSpecialCharactersLinq(input);
        }
        sw.Stop();
        Console.WriteLine($"LINQ: {sw.ElapsedMilliseconds} ms");

        // Test StringBuilder method
        sw.Restart();
        char[] specialChars = { '@', '#', '$', '%', '^', '&', '*', '|', '_' };
        for (int i = 0; i < iterations; i++)
        {
            StringBuilderCharacterRemoval.RemoveCharactersStringBuilder(input, specialChars);
        }
        sw.Stop();
        Console.WriteLine($"StringBuilder: {sw.ElapsedMilliseconds} ms");
    }
}

Complete Example Program

using System;

class UnicodeStringCleaner
{
    static void Main(string[] args)
    {
        Console.WriteLine("=== Unicode String Special Character Removal ===\n");

        // Test all methods
        SimpleCharacterRemoval.TestSimpleRemoval();
        Console.WriteLine();
        
        RegexCharacterRemoval.TestRegexMethods();
        Console.WriteLine();
        
        LinqCharacterRemoval.TestLinqMethods();
        Console.WriteLine();
        
        StringBuilderCharacterRemoval.TestStringBuilderMethods();
        Console.WriteLine();
        
        AdvancedUnicodeHandling.TestAdvancedMethods();
        Console.WriteLine();

        // Performance comparison
        PerformanceComparison.BenchmarkMethods();

        // Interactive mode
        InteractiveMode();
    }

    static void InteractiveMode()
    {
        Console.WriteLine("\n=== Interactive Mode ===");
        Console.WriteLine("Enter text to clean (or 'exit' to quit):");

        while (true)
        {
            Console.Write("\nInput: ");
            string input = Console.ReadLine();

            if (string.IsNullOrWhiteSpace(input) || input.ToLower() == "exit")
                break;

            Console.WriteLine("\nChoose cleaning method:");
            Console.WriteLine("1. Remove specific characters (@#$%^&*|_)");
            Console.WriteLine("2. Keep only alphanumeric");
            Console.WriteLine("3. Keep Unicode letters and numbers");
            Console.WriteLine("4. Remove punctuation");
            Console.Write("Choice (1-4): ");

            if (int.TryParse(Console.ReadLine(), out int choice))
            {
                string result = choice switch
                {
                    1 => RegexCharacterRemoval.RemoveSpecialCharacters(input),
                    2 => RegexCharacterRemoval.KeepAlphanumericOnly(input),
                    3 => RegexCharacterRemoval.KeepUnicodeLettersAndNumbers(input),
                    4 => RegexCharacterRemoval.RemovePunctuation(input),
                    _ => "Invalid choice"
                };

                Console.WriteLine($"Result: {result}");
            }
        }

        Console.WriteLine("Thanks for using the Unicode String Cleaner!");
        Console.ReadKey();
    }
}

Best Practices

Choose the Right Method:
- String.Replace: For few specific known characters
- Regex: For pattern-based removal and Unicode support
- StringBuilder: For large strings or frequent operations
- LINQ: For functional-style programming and readability
Unicode Considerations:
- Use \p{L} and \p{N} in regex for Unicode letter/number matching
- Consider normalization for accented characters
- Be aware of surrogate pairs for characters outside the Basic Multilingual Plane
Performance Tips:
- Compile regex patterns for repeated use
- Use StringBuilder for building large strings
- Consider caching results for frequently processed strings
Security Considerations:
- Always validate input length to prevent DoS attacks
- Be explicit about what characters you allow vs. remove
- Consider using allowlists instead of blocklists when possible

Common Use Cases

Data Sanitization: Cleaning user input for database storage
SEO-Friendly URLs: Removing special characters from page titles
File Naming: Creating valid file names from user input
Search Indexing: Normalizing text for search functionality
Data Import/Export: Cleaning data between different systems

Conclusion

Removing special characters from Unicode strings requires understanding your specific requirements and choosing the appropriate method. Whether you need simple character replacement or complex Unicode normalization, C# provides powerful tools to handle text processing efficiently. Consider performance requirements, internationalization needs, and maintainability when selecting your approach.

Twitter Facebook LinkedIn

Mahedee Hasan

Removing Special Characters from Unicode Strings in C#: Complete Guide

Understanding Unicode and Special Characters

Method 1: Simple Character Replacement

Method 2: Regular Expression Approach

Important Regex Patterns:

Method 3: LINQ-Based Approach

Method 4: StringBuilder for Performance

Method 5: Advanced Unicode Handling

Performance Comparison

Complete Example Program

Best Practices

Common Use Cases

Conclusion

Comments

You May Also Enjoy

Implementing Distributed Transactions in Microservices: SAGA Pattern with RabbitMQ and ASP.NET Core

Middleware in ASP.NET Core: Why It Matters and How to Build Custom Middleware with a Real-World Examples

Factory Design Pattern Explained with 10 Real-World Use Cases

Factory Design Pattern using C# with a real world banking example