This post focuses on practical details and what they mean for your day-to-day development, with an eye toward where we’re headed next.<\/p>\n\n\n\n
In our previous article<\/a>, we announced the USTRING data type was coming back, and its intended role in Clarion 12’s Unicode support. Now, let’s explore the implementation details that will help you work more effectively with the USTRING.<\/p>\n\n\n\n At its core, the USTRING data type uses UTF-16 encoding, allocating two bytes per character. This architectural decision provides several key advantages:<\/p>\n\n\n\n When you declare Important:<\/strong> The null terminator WCHAR(0) occupies 2 bytes because it’s a wide character, just like every other character in the string. This is how functions like One of USTRING’s practical strengths is transparent handling of both Unicode and ANSI content. You can freely mix Unicode literals and ANSI strings in your code:<\/p>\n\n\n\n The runtime handles conversions automatically, respecting the current code page settings. When working with international applications, you can set the code page and locale to ensure proper character handling:<\/p>\n\n\n\n This is where developers first encounter USTRING’s two-byte nature. The distinction between LEN()<\/strong> returns the logical length\u2014the number of characters actually stored in the string. This is what you typically care about when processing text.<\/p>\n\n\n\n SIZE()<\/strong> returns the allocated byte capacity. For a Understanding this distinction matters when:<\/p>\n\n\n\n For fixed-size declarations like This is why SIZE() is so fast: it’s just a constant value, not a calculation that happens when your code runs.<\/p>\n<\/div>\n\n\n\n When you declare a USTRING, the actual memory allocated is double the character count you specify:<\/p>\n\n\n\n Choose appropriate sizes to avoid wasting memory. Here’s what oversizing costs:<\/p>\n\n\n\n Understanding when to worry about USTRING memory overhead:<\/p>\n\n\n\n Rule of thumb:<\/strong> For individual strings, use generous sizes. For large queues, arrays, or tables, size more carefully.<\/p>\n\n\n\n Design-time declarations have a maximum size of 4MB, while runtime allocations can be sized dynamically based on your application’s needs.<\/p>\n\n\n\n When initializing or assigning to a USTRING, use the You can read individual characters using slice syntax, but cannot assign to them:<\/p>\n\n\n\n This restriction maintains string integrity in the UTF-16 implementation.<\/p>\n\n\n\n The current implementation doesn’t support Unicode strings in these specific contexts:<\/p>\n These are implementation-specific constraints that may be addressed in future releases.<\/p>\n<\/div>\n\n\n\n When you write a string expression like this:<\/p>\n\n\n\n The Clarion runtime evaluates it using a string stack\u2014a temporary workspace for building the final result. Here’s the step-by-step process:<\/p>\n\n\n\n This stack-based approach doesn’t create temporary variables that need cleanup. The runtime handles all intermediate strings automatically, and they vanish when the expression completes.<\/p>\n\n\n\n Why this matters for you:<\/strong><\/p>\n\n\n\n <\/p>\n\n\n\n The string stack is why expressions like Best practice:<\/strong> Write natural, readable string expressions. The runtime is optimized for this pattern.<\/p>\n<\/div>\n\n\n\n When moving existing ANSI string code to USTRING:<\/p>\n\n\n\n The USTRING implementation provides a solid foundation for Unicode support while maintaining the Clarion language’s characteristic simplicity. The UTF-16 encoding, dual character set support, and clear LEN\/SIZE distinction give you the tools needed for modern, international applications.<\/p>\n Key takeaways:<\/strong><\/p>\n Thanks for being part of the Clarion community. If you try this out, let us know what you think \u2014 and stay tuned, there’s more to come.<\/em><\/p>\n\n\n\n Related:<\/strong> Clarion 12 Beta: USTRING Returns ANSI & Unicode<\/a><\/p>\n","protected":false},"excerpt":{"rendered":" This post focuses on practical details and what they mean for your day-to-day development, with an eye toward where we’re headed next. In our previous article, we announced the USTRING data type was coming back, and its intended role in Clarion 12’s Unicode support. Now, let’s explore the implementation details that will help you work … Continue reading Understanding USTRING: A Deep Dive into Clarion 12’s UTF-16 Implementation<\/span> USTRING is UTF-16: What This Means for You<\/h2>\n\n\n\n
\n
Name USTRING(21) ! 21-character Unicode string\nCompany USTRING('SoftVelocity') ! Initialized with a value\nPhone USTRING(@P(###)###-####P) ! Formatted with picture token\nMyStr USTRING(20) ! 20 characters available\n<\/code><\/pre>\n\n\n\nUSTRING(20)<\/code>, you’re reserving space for 20 characters plus a null terminator. Internally, this allocates 42 bytes (21 characters \u00d7 2 bytes each).<\/p>\n\n\n\nMemory Layout: USTRING(20)\n\nDeclaration: USTRING(20)\nAllocation: [ 40 bytes for 20 characters ][2 bytes null]\n |<\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 20 chars \u00d7 2 bytes \u2500\u2500\u2500\u2500\u2500>|\nTotal Size: 42 bytes\n\nExample with \"Hello\":\nPosition: 1 2 3 4 5 6-20 21\nCharacter: H e l l o (empty) \\0\nBytes: [H ][e ][l ][l ][o ][ ... ][\\0]\n 2 2 2 2 2 30 2\n WCHAR(0)\n <\u2500\u2500\u2500 40 bytes for data \u2500\u2500\u2500\u2500> <2>\n\nTotal: 42 bytes allocated (40 for characters + 2 for null terminator)\n<\/code><\/pre>\n\n\n\nlstrlenW<\/code> know where the string ends\u2014they scan for this 2-byte null value.<\/p>\n\n\n\nDual Character Set Support: The Best of Both Worlds<\/h2>\n\n\n\n
MYUSTR USTRING(50)\n\nCODE\n MYUSTR = u'\u0391 \u03a9' ! Greek Unicode characters\n MYUSTR = 'Regular Text' ! ANSI text works too\n MYUSTR = u'Mix: ' & '\u0391 \u03a9' ! Concatenate both types\n<\/code><\/pre>\n\n\n\nSYSTEM {PROP:Codepage} = 1253 ! Greece\nSYSTEM {PROP:Locale} = 1032 ! Greece\n<\/code><\/pre>\n\n\n\nLEN() vs SIZE(): A Critical Distinction<\/h2>\n\n\n\n
LEN()<\/code> and SIZE()<\/code> directly reflects the UTF-16 implementation:<\/p>\n\n\n\nMYUSTR USTRING(20)\nL LONG\nS LONG\n\nCODE\n MYUSTR = u'\u0391 \u03a9' ! 3 Unicode characters\n L = LEN(MYUSTR) ! L = 3 (character count)\n S = SIZE(MYUSTR) ! S = 40 (20 characters \u00d7 2 bytes)\n<\/code><\/pre>\n\n\n\nUSTRING(20)<\/code>, SIZE() always returns 40, regardless of how many characters you’ve stored. This represents the maximum storage available.<\/p>\n\n\n\n\n
How SIZE() Actually Works<\/h3>\n
USTRING(20)<\/code>, SIZE() is calculated by the Clarion compiler at compile-time<\/strong>. The compiler knows the capacity is 20 characters and generates code that returns 20 \u00d7 2 = 40<\/code> directly\u2014no runtime function call needed.<\/p>\nWhen LEN() and SIZE() Differ: A Practical Example<\/h3>\n\n\n\n
UserInput USTRING(100) ! Allocated capacity: 100 chars\nBytes LONG\nChars LONG\n\nCODE\n UserInput = '' ! Empty string\n Chars = LEN(UserInput) ! Chars = 0 (no content)\n Bytes = SIZE(UserInput) ! Bytes = 200 (capacity still allocated)\n\n UserInput = u'Hi' ! Short string\n Chars = LEN(UserInput) ! Chars = 2 (actual content)\n Bytes = SIZE(UserInput) ! Bytes = 200 (capacity unchanged)\n\n ! Key insight: SIZE() never changes after declaration\n ! LEN() reflects actual content\n<\/code><\/pre>\n\n\n\nMemory Allocation: Understanding the 2:1 Ratio<\/h2>\n\n\n\n
Small USTRING(10) ! Allocates 20 bytes (10 \u00d7 2)\nMedium USTRING(100) ! Allocates 200 bytes (100 \u00d7 2)\nLarge USTRING(1000) ! Allocates 2000 bytes (1000 \u00d7 2)\n<\/code><\/pre>\n\n\n\nRight-Sizing Your Strings<\/h3>\n\n\n\n
! Good - sized appropriately\nFirstName USTRING(50) ! 100 bytes allocated\n\n! Wasteful - unnecessarily large\nUserName USTRING(500) ! 1000 bytes allocated\n ! If only ~50 chars used: 100 used, 900 wasted\n\nComment USTRING(5000) ! 10,000 bytes allocated\n ! If only ~100 chars used: 200 used, 9,800 wasted\n<\/code><\/pre>\n\n\n\nWhen Memory Size Actually Matters<\/h3>\n\n\n\n
! Scenario 1: Single string - overhead is negligible\nCustomerName USTRING(100) ! 200 bytes total\n! Impact: Minimal - 100 extra bytes compared to ANSI\n\n! Scenario 2: Large collections - overhead multiplies\nCustomerQueue QUEUE\nName USTRING(100) ! 200 bytes\nAddress USTRING(200) ! 400 bytes\nCity USTRING(50) ! 100 bytes\n END\n\n! Impact with 100,000 records in queue:\n! USTRING: 70,000,000 bytes (70 MB)\n! STRING: 35,000,000 bytes (35 MB)\n! Difference: 35 MB - this is where sizing matters!\n\n! Or with an array:\nCustomerArray USTRING(100), DIM(100000) ! 20,000,000 bytes (20 MB)\n! vs STRING(100), DIM(100000) ! 10,000,000 bytes (10 MB)\n<\/code><\/pre>\n\n\n\nDesign-Time vs Runtime Allocation<\/h3>\n\n\n\n
! Design-time: Fixed size declared in source\nMyStr USTRING(100) ! 200 bytes allocated at compile time\n\n! Runtime: Dynamic allocation with NEW\nMyStr &USTRING ! Reference to dynamically allocated string\nCODE\n MyStr &= NEW USTRING(100) ! 200 bytes allocated at runtime\n<\/code><\/pre>\n\n\n\nWorking with Unicode Literals<\/h2>\n\n\n\n
U<\/code> or u<\/code> prefix for Unicode literals:<\/p>\n\n\n\nMyStr USTRING(50)\nCODE\n MyStr = U'\u03a9 \u03b1 \u03b2' ! Correct - U prefix for Unicode\n MyStr = u'\u03a9 \u03b1 \u03b2' ! Also correct - lowercase works too\n MyStr = '\u03a9 \u03b1 \u03b2' ! Works but may not preserve Unicode properly\n<\/code><\/pre>\n\n\n\nPractical Implications for Your Code<\/h2>\n\n\n\n
Character Access is Read-Only on Assignment<\/h3>\n\n\n\n
C = MyStr[5] ! Read character at position 5 - ALLOWED\nMyStr[1] = 'A' ! ERROR - Not allowed, creates invalid string\n<\/code><\/pre>\n\n\n\nUse LEN() for Logic, SIZE() for Memory<\/h3>\n\n\n\n
! Correct usage\nIF LEN(UserInput) > 0 ! Check if string has content\n ! Process input\nEND\n\n! Memory allocation calculation\nBytesNeeded = SIZE(MyStr) ! Get total allocated bytes\n<\/code><\/pre>\n\n\n\nCurrent Limitations<\/h3>\n
\n
EVALUATE<\/code> statement<\/li>\nMATCH<\/code> built-in function<\/li>\nSTRPOS<\/code> built-in function<\/li>\n<\/ul>\nWorking Example: Practical USTRING Usage<\/h2>\n\n\n\n
MAP\n MODULE('API')\n GetSystemInfo(*LONG, *LONG), PROC, RAW, PASCAL, NAME('GetSystemInfo')\n END\nEND\n\nMyName USTRING(50)\nMyCompany USTRING(100)\nFullInfo USTRING(200)\nCharCount LONG\nByteCount LONG\n\nCODE\n ! Assign Unicode content\n MyName = u'\u0391\u03bb\u03ad\u03be\u03b1\u03bd\u03b4\u03c1\u03bf\u03c2' ! Greek name\n MyCompany = u'SoftVelocity' ! Company name\n \n ! Concatenate strings\n FullInfo = MyName & u' - ' & MyCompany\n \n ! Get character count and byte size\n CharCount = LEN(FullInfo) ! Actual characters in string\n ByteCount = SIZE(FullInfo) ! Total bytes allocated\n \n ! Display results\n MESSAGE('Name: ' & MyName & |\n '|Characters: ' & CharCount & |\n '|Bytes Allocated: ' & ByteCount)\n<\/code><\/pre>\n\n\n\nBehind the Scenes: What Happens When You Concatenate<\/h2>\n\n\n\n
Result = FirstName & ' ' & LastName\n<\/code><\/pre>\n\n\n\nString Expression Evaluation\n\nStep 1: Push FirstName onto stack \u2192 Stack: [FirstName]\nStep 2: Push ' ' onto stack \u2192 Stack: [FirstName][' ']\nStep 3: Concatenate top 2 items \u2192 Stack: [FirstName ]\nStep 4: Push LastName onto stack \u2192 Stack: [FirstName ][LastName]\nStep 5: Concatenate top 2 items \u2192 Stack: [FirstName LastName]\nStep 6: Pop result into Result variable \u2192 Result gets final string\n<\/code><\/pre>\n\n\n\n\n
Performance Implication<\/h3>\n
Name = FirstName & ' ' & MiddleName & ' ' & LastName<\/code> don’t create memory leaks or slow down your application. Each intermediate result (FirstName & ' '<\/code>, etc.) exists only temporarily on the stack and is automatically cleaned up.<\/p>\nCommon Pitfalls and How to Avoid Them<\/h2>\n\n\n\n
Pitfall 1: Using SIZE() When You Mean LEN()<\/h3>\n\n\n\n
! WRONG - This won't work as expected\nName USTRING(50)\nCODE\n Name = u'John'\n IF SIZE(Name) > 10 ! Always TRUE (SIZE is 100, not 8)\n ! This always executes\n END\n\n! CORRECT - Use LEN() for content checks\n IF LEN(Name) > 10 ! FALSE (LEN is 4)\n ! This executes only when needed\n END\n<\/code><\/pre>\n\n\n\nPitfall 2: Buffer Size Confusion<\/h3>\n\n\n\n
! WRONG - Allocating based on character count for bytes\nName USTRING(50)\nBuffer STRING(LEN(Name)) ! Too small! Only 50 bytes, need 100\n\n! CORRECT - Use SIZE() for byte allocations\nBuffer STRING(SIZE(Name)) ! Correct: 100 bytes\n<\/code><\/pre>\n\n\n\nPitfall 3: Forgetting the U Prefix<\/h3>\n\n\n\n
Greek USTRING(20)\nCODE\n ! INEFFICIENT - ANSI string converted to Unicode at runtime\n Greek = '\u0391\u03b8\u03ae\u03bd\u03b1'\n\n ! EFFICIENT - Direct Unicode assignment, no conversion\n Greek = u'\u0391\u03b8\u03ae\u03bd\u03b1'\n<\/code><\/pre>\n\n\n\nMigration from STRING to USTRING: Real Examples<\/h2>\n\n\n\n
Example 1: Buffer Sizing<\/h3>\n\n\n\n
! Before (ANSI STRING)\nName STRING(50) ! 50 bytes\nBuffer STRING(SIZE(Name)) ! 50 bytes\n\n! After (USTRING)\nName USTRING(50) ! 100 bytes (50 \u00d7 2)\nBuffer STRING(SIZE(Name)) ! 100 bytes - SIZE() handles it correctly\n<\/code><\/pre>\n\n\n\nExample 2: Loop Iterations<\/h3>\n\n\n\n
! Before (ANSI STRING)\nText STRING(100)\nI LONG\nCODE\n LOOP I = 1 TO LEN(Text) ! Good - use LEN() not SIZE()\n ! Process Text[I]\n END\n\n! After (USTRING)\nText USTRING(100)\nI LONG\nCODE\n LOOP I = 1 TO LEN(Text) ! Same - LEN() still correct\n ! Process Text[I]\n END\n ! KEY: LEN() works the same way for both types!\n<\/code><\/pre>\n\n\n\nExample 3: API Calls<\/h3>\n\n\n\n
! Before (ANSI STRING)\nBuffer STRING(1000)\nSize LONG\nCODE\n Size = SIZE(Buffer) ! 1000 bytes\n ! Pass Size to Windows API expecting byte count\n\n! After (USTRING)\nBuffer USTRING(1000)\nSize LONG\nCODE\n Size = SIZE(Buffer) ! 2000 bytes (1000 \u00d7 2)\n ! SIZE() correctly returns byte count for Unicode APIs\n<\/code><\/pre>\n\n\n\nMigration Considerations<\/h2>\n\n\n\n
\n
LEN()<\/code> for character-based logic, not SIZE()<\/code><\/li>\n\n\n\nU<\/code> prefix to string literals containing Unicode characters<\/li>\n\n\n\nEVALUATE<\/code>, MATCH<\/code>, and STRPOS<\/code> limitations<\/li>\n\n\n\nLooking Forward<\/h2>\n
\n
\n\n\n\n