Adding support for UTF8 Files and Folders names [Joomla 3.x]

[Makulia]

Hello, Jan!

I have made and tested the path modifications for the Phoca Gallery to make it work with UTF8(multibyte) file|folder names.

I know that some users requested this feature long time ago!

Problem

Phoca Gallery is not allowing to create and display (in some cases) folders using your native language (expet English) plus limit some allowes special chars in file names like №.

Solution

It is not good that in the 21 sentry we need some excuses not to support utf8 file and folders names (Wikimedia does it for a long time!).
First, I love Phoca products, especially Gallery, and I have decided to make some changes and fix this limitation!

1) Setlocale before using basename! (it is requed accroding to php.net manual)

Insert setlocale(LC_ALL, 'C.UTF-8', 'C'); in all files, where you are using basename!

Like:

Code: Select all

defined( '_JEXEC' ) or die( 'Restricted access' );
jimport( 'joomla.filesystem.folder' ); 
jimport( 'joomla.filesystem.file' );
phocagalleryimport('phocagallery.image.image');
phocagalleryimport('phocagallery.path.path');
phocagalleryimport('phocagallery.file.filefolder');
setlocale(LC_ALL, 'C.UTF-8', 'C');

2) Use new Joomla 3.x api + replays getCMD with getString to get full file name!

Edit administrator/components/phoca_gallery/controllers/phocagelleryu.php

Code: Select all

$phocaGalleryApp = JFactory::getApplication();
		$folderNew      = $phocaGalleryApp->input->getSTRING('foldername', '');
		$folderCheck    = $phocaGalleryApp->input->getSTRING('foldername', null, '', 'string', JREQUEST_ALLOWRAW);
		//$folderNew		= JRequest::getCmd( 'foldername', '');
		//$folderCheck	= JRequest::getVar( 'foldername', null, '', 'string', JREQUEST_ALLOWRAW);

I have tested this changes with russian file|folder names and it works like a charm!

Could you please insert this update to the next PG release!?

If you interested in some details, please PM me.

[Jan]

Hi, thank you very much for the guide, I have added it to feature request list and will take a look at it.

Does the getCmd not work with utf-8?

Thank you, Jan

[Makulia]

Let me explain the difference:

Code: Select all

 CMD
// Allow a-z, 0-9, underscore, dot, dash. Also remove leading dots from result. 
$result = (string) preg_replace('/[^A-Z0-9_\.-]/i', '', $source);
$result = ltrim($result, '.');

while

Code: Select all

STRING
// Converts the input to a plain text string; strips all tags / attributes.
$result = (string) $this->_remove($this->_decode((string) $source));

So when you are using cmd, you simply limit a number of chars available.

Could you point me to that list?

I have found old post about this problem. My solution solves this problem for any language.

[Jan]

Hi, thank you for the info, I just thought about security issues when changing cmd to string (for example removing tags or converting them to the html entities, etc )

Jan

[Makulia]

First, it is already

strips all tags / attributes

Second, I have tried to create folder with name like

Code: Select all

<? php info();

Folder is created, but with name

Code: Select all

?php info();

(Linux server)

So unsupported symbols were removed. Seems not to much a security problem.

When the copying file to Windows via sftp, name is changed to, because ? is the restricted symbol in Windows.

Code: Select all

%3Fphp info();

Of course, we can write simple regexp to remove unsupported in File|folder names symbols from it's name!

Can use for reference Filename#Reserved_characters_and_words in wiki.
As for Windows, this symbols are prohibited for use in file|folder names:

Code: Select all

\/:*?"<>|

So we can trim them from entered folder name or replace with _ symbol. And that would be enough. What do you think?

[Jan]

Hi, thank you for the info. Yes, the problem by open source code is, you never know on which environment the script will be running - this is why there is a cmd filtering in Joomla! - to prevent from problems on different server environments - there are many types: windows, linux, unix, freebsd, not sure if on mac you can run apache and php, but maybe too. and a lot of different settings of php, apache, mysql, etc. So this is why the standard Joomla! behaviour was set as you never know where the user will run it. And it needs to be tested because of security reason - so it is not easy to add it as when we add it, we need to test it everywhere we can - as adding something to URL must be preciously checked and properly treated.

Jan

[Makulia]

Yes, you can run it on mac too!
Cmd is one of the many filters in Joomla, but it is not created to be the barrier for internationalisation.
As we are talking of internationalisation support here, Joomla itself, for example, has support for utf8 sef urls in any language out of the box!

Yep, there are many platforms, but there also are code writing standards. For example, you are using basename, but doesn't set locale before, but you must do it according php manual! So basename can not determine file encoding correctly.
If we stick to them, our code is interpreted well on any platform.
In this case, correctly would be to say, that we are talking not about open source code (it may be closed, doesn't make any difference!) but about cross platform software. And main benefit of it - you code once and it runs on any supported platforms.

Internationalisation of file|folders names in php does not affect security in any way! It also does not depends on platform, you run it at! All the allowed characters in file|folder names are publicly know (you can see them in wiki, for example). If we want to make sure, no special chars passed, as I say, we can write simple regex check. Also, for better security of form submission, I strongly recommend using prepared statements.

[Jan]

Hi, added in 4.2.0

Jan

[Makulia]

Hi, Jan! Thank you for the great work! I am very happy to hear it! Will definitely check it out.