3.0.0

Base Distribution

[TASK] Remove TYPO3.Party from set-dependencies.sh

[TASK] Remove unintentional typo3/party requirement

[TASK] Update PHPUnit requirement to 4.5.*

Updates PHPUnit to version 4.5.*

[TASK] Update PhpUnit to 4.3 and vfsStream to 1.4

[TASK] Update PHPUnit and vfsStream dependencies

This updates PHPUnit from 4.5 to 4.6 and vfsStream from 1.4 to 1.5.

[TASK] Fix dependencies to branch versions

[TASK] Adjust to changed parameters for commit_manifest_update

Related: NEOS-1216 Depends: I791174633d2b1b0fb7263504744901d87bbb1ec6

[TASK] Do not require dev-master of doctrine/migrations

With doctrine/migrations not having had a stable branch for years, Flow always required dev-master. Now that 1.0 is being worked on, require @dev instead and delegate the version to the Flow package.

Related: FLOW-300

[TASK] Add flowpack/behat to distribution

We have Behat tests in the Flow package, but the base distribution lacks the needed tooling to easily run them. With this change, you can do this:

cd Build/Behat
composer install

When this is finished, running the tests is possible:

bin/behat --ansi -f progress -v -c Packages/Framework/TYPO3.Flow/Tests/Behavior/behat.yml.dist

[BUGFIX] Use more specific version of doctrine/migrations

This pins the version to a release candidate (~1.0.0 is specified in TYPO3.Flow) instead of the dev-master version that is implied by “@dev”.

TYPO3.Eel

[BUGFIX] Fix unit test in ExpressionSyntaxValidatorTest test case

Fixes a typo in a unit test introduced with Iaec2e11dfac0b35510c9fc0f6ac08e7d80d92268.

[FEATURE] Provide an Eel expression syntax validator

This adds a Flow Validator which allows to check for a correct syntax. The expression given must lack the wrapping ${…}.

The validation is done by simply handing the given expression to the eel parser.

[TASK] Mark recent core migrations applied

This commit doesn’t contain any changes, it simply marks recent migrations applied so that:

./flow flow:core:migrate --status

won’t show any open migrations for this package.

[FEATURE] Add SecurityHelper

Adds a helper for security related information and checks. This implementation is just a stub for now to help fix a bug. The only available method is getAccount() more should be added in the future.

Related: NEOS-1012

[FEATURE] Additional Fizzle comparison operators (<, <=, >, >=)

Implements lessThan, lessThanOrEqual, greaterThan, greaterThanOrEqual comparison operators in Fizzle.

Example usage:

{q(site).children('someNodesInHere').filter('[someProperty >= 10]')}

Related: FLOW-178

[BUGFIX] Children operation does not work with collections

Children operation applied to entities and giving the name of a collection property don’t retrieve the collection

[FEATURE] Add String.length(s) helper function

Resolves: FLOW-141

[TASK] Support for non-wrapped value in FlowQuery add() operation

Implements support to add values to a FlowQuery result without wrapping:

q(node).add(someOtherNode)

Traversable arguments are still supported:

q(node).add(q(someOtherNode))

Resolves: FLOW-142

[BUGFIX] children() operation with an empty context should never fail

This change makes sure that the children operation always returns an empty result if an empty context was given. This is needed because the TYPO3CR children operation might be chained and the later operation will be resolved to the object version.

Fixes: NEOS-523

[TASK] Make cached Eel expression avoid duplicate declarations

After the introduction of the EntityPrivilegeExpressionEvaluator with the ACL changes the functional ActionControllerTest failed with:

Cannot redeclare expression_c4cc6f98eb99414122ca432a08debb4c()

By wrapping the cached Eel expression code in function_exists() checks this error is avoided.

[FEATURE] Add isFloat, isObject & isScalar to TypeHelper

Adds three additional functions to the TypeHelper, which were left out in the first implementation.

[FEATURE] Add TypeHelper to get variable type information

This helper allows to get information about variable types in EEL expressions.

[BUGFIX] EEL expressions should be trimmed before parsing

As the EEL parser will not accept expressions that start or end with a whitespace the evaluators should trim given expressions to avoid hard to spot errors due to additional whitespaces.

Fixes: FLOW-278

[TASK] Correct indentation in EelExpressionRecognizer

[BUGFIX] Nested object literals not supported

The parser does not support nested object literals like:

{bar: {foo: 'baz'}}

but only object literals containing literal values.

Fixes: FLOW-314

[FEATURE] Path filter for Fizzle grammar

Until now only object identifiers or property names (both without slashes) could be matched. This change adds support for absolute and relative paths.

[TASK] Add I18n Eel helper to provide translation methods

The Eel helper provides 3 different entry points.

  • By calling translate, you can pass all options at once
  • By calling translate with a shorthand string (PackageKey:Source:trans-unit-id), you can pass all options at once
  • By calling id or value, an object is created, that allows you to collect these options via method chaining.

The object for collection translation options implements a __toString method, so that there is no need for a finishing method call.

[BUGFIX] Fix two unknown class usages

Fixes two places where Exception was used and did not resolve to a class.

(cherry picked from commit bc94d1283f6696a929af5f6764d2227508cbcc9c)

[TASK] Remove I18n helper again

This basically reverts commit 728177b1d6ebd51fb6203da65a074aee90182493.

Functionality has been moved to the existing Flow TranslationHelper to avoid duplicate helpers and a hard dependency on Flow in EEL.

[BUGFIX] Return simple string when id provided isn’t a i18n label

When using the I18nHelper and providing only the id argument, it should return the string as is if it’s not a valid label id.

(cherry picked from commit 26cd2f7e8c00366a8cb407ef86bf76976dfd400b)

[BUGFIX] Only set locale in I18nHelper if not NULL

When providing a single argument (id) to the translate method the helper tries to set the locale in the translation parameter token to NULL, which results in an exception.

(cherry picked from commit fe411b4dcd348828fb4c89729878833afc001f1a)

TYPO3.Flow

[TASK] Add new files to default .gitignore file

The default .gitignore file in Flow needs to contain the (new) rST files that replace Readme.txt and Upgrading.txt.

[TASK] Undo requirements adjustment from CI job

The release CI job adjusted requirements dutifully, but the result was not as expected, because we humans missed some needed changes.

[TASK] Remove unintentional typo3/party requirement

[TASK] Add changelog for TYPO3 Flow 3.0.0-beta1

See https://ci.neos.typo3.org/job/typo3-flow-release/30/

[BUGFIX] Make proxy compilation possible without DB connection

A hotfix to make proxy compilation work when no DB connection is possible. This used to work, but now a connection to the DB is needed to compile Doctrine proxies.

This is probably a side effect of the Doctrine update.

Related: FLOW-219

[BUGFIX] Fix tests that failed when TYPO3.Party is not installed

This change fixes some unit and functional tests that failed if the TYPO3.Party package was not active.

Background:

With Flow 3.0 the Party package is no longer part of the base distribution. The AbstractParty type hint in Account::setParty() leads to invalid proxy classes because the doctrine proxy builder will throw away the type hint if it can’t be resolved. Furthermore this adjusts the unit tests for the Account class so that the mocks don’t depend on the non-existing AbstractParty.

Related: FLOW-5

[TASK] Set FLOW_VERSION_BRANCH to 3.0 in Bootstrap

[TASK] Update Readme and Upgrading instructions for 3.0 release

[TASK] Add missing changelogs (for 2.x)

[TASK] Mark recent core migrations applied

This commit doesn’t contain any changes, it simply marks recent migrations applied so that:

./flow flow:core:migrate --status

won’t show any open migrations for this package.

[BUGFIX] Fix error and standard view templates

Fixes several minor issues in regards of rendering of the default Error and StandardView` templates:

  • Inline “StandardView_FloatingWindow.png” background image (it wasn’t loaded otherwise when not in web root because of missing base tag)
  • Remove references to “StandardView_Package.png” that doesn’t exist
  • Commit: febbfcd

[TASK] Adjust one unit test to PHP 7

One of the unit tests fails under PHP 7, because the signature of DateTime::createFromFormat() has changed.

This test adjusts that while keeping BC.

[BUGFIX] Rename redirectToReferringRequest() to forwardToReferringRequest() in ActionController

ActionRequest::redirectToReferringRequest() actually triggers a forward not a redirect. This change deprecates the redirectToReferringRequest() method in favor of a new method redirectToReferringRequest() which works exactly like the previous method.

The deprecated method might be re-implemented to issue a “real” HTTP redirect at some point, so you should not use it for forwards.

Fixes: FLOW-164

[BUGFIX] Enable Functional Tests for PSR-4 packages

Packages with PSR-4 autoloading will not be able to work with functional tests because two problems arise. First the package namespace is prepended twice and second the path contains an unnecessary backslash.

To avoid wrong amounts of backslashes we now use the path merging utility to merge namespace parts instead of doing simple string concatenation. Additionally a check is added for packages with PSR-4 autoloading enabled.

Releaes: master, 1.2

[FEATURE] Reconnect the EntityManager automatically

If the persistence backend closes the connection in the background. E.g. if the MySQL server is configured with a low wait_timeout setting of 10-30 seconds. The PersistenceManager now catches the exception, establishes a new connection automatically and tries to send all entity manager changes to the backend again. This is really important for long running commands or tasks.

Without this central improvement in the persistence manager it’s really hard to build stable daemon workers based on Flow commands.

Resolves: FLOW-207

[TASK] Improve error message in convertObjectToIdentityArray

This change gives some hint about the object in question when trying to convert it into an identity array but failing because it is unknown to the persistence manager.

[BUGFIX] Schema validation: Accept interfaces for format “class-name”

This change adjusts the SchemaValidator to accept strings referring interfaces even if format “class-name” is required.

Background:

Currently if a string is expected to have the format “class-name”, like:

{ type: string, format: class-name }

in a schema validation, only actual class names are accepted. To match interfaces the “interface-name” format can be used.

In Flow we often use interface names in Objects.yaml in order to keep things decoupled and flexible. Because it’s not possible to specify multiple supported formats, this relaxes this restriction.

[FEATURE] Human friendly error message in cache:flushone

This change introduces a human friendly error message for the cache:flushone command, if the user entered a non-existing cache identifier.

Instead of showing an uncaught exception, the user will now be provided with a hint:

The cache "TYPO3.TypoScript.Content" does not exist.
Did you mean "TYPO3_TypoScript_Content"?

[TASK] Tweak “skip detection” in two schema migrations

Uses skipIf() instead of a plain if with an empty return.

[BUGFIX] ObjectArray should use TypeHandling::getTypeForValue

ObjectArray used get_class() to get the type of given $value. This change changes that to using TypeHandling::getTypeForValue.

This fixes problems where for example class names of doctrine proxies are used instead of the entity class name.

Resolves: FLOW-206

[BUGFIX] Support numeric identifiers in dynamic route parts

Currently the dynamic route part handler provided by Flow (namely DynamicRoutePart and IdentityRoutePart) only support objects with string identifiers.

With this change also numeric identifiers are supported, for example for entities with a numeric auto_increment identifier.

[BUGFIX] Adjust our Doctrine\Service to DBAL 2.4

The internals of ForeignKeyConstraint changed between DBAL 2.3 and 2.4, so this adjusts our tweaking of internal state to fit.

Related: FLOW-198

[FEATURE] HHVM compatibility

This commit introduces HHVM compatibility by working around some behavioural differences between vanilla PHP and HHVM.

Fixes: FLOW-194

[!!!] Decoupling of TYPO3.Party package

This makes the use of the TYPO3.Party package optional, hence fully replaceable. The Account does not have a tight coupling to the AbstractParty Aggregate Root anymore, so complete customized solutions are usable now.

This change deprecates the following methods:

  • Account::getParty()
  • Account::setParty()
  • Security\\Context::getParty()
  • Security\\Context::getPartyByType()

They still work if the party package is installed, but usage of their methods should be replaced with custom service calls (see party package for an example of a simple PartyService).

This is a breaking change because it removes the dependency to the TYPO3.Party package. If a package makes use of that package (e.g. by extending the AbstractParty model, an entry:

"typo3/party": "~3.0"

in the “require” section of the composer.json file is to be added! Besides this change adjusts the getParty() and setParty() methods of Account and rearranges the database structure; so in cases where these changes might influence userland code, adjustments might be necessary.

Fixes: FLOW-5

[FEATURE] Account::isActive()

This adds a convenience method to Account which allows for checking if the account is currently active.

Also marks the Account class and its method as API.

Related: NEOS-962

[BUGFIX] Fix broken unit test due to withoutAuthorizationChecks

The call to withoutAuthorizationChecks cannot be easily mocked, therefor the test prevents mocking of that method and mocks only the necessary methods.

[BUGFIX] Remove duplicate use statement

This change removes a duplicate use statement for the UnitTestCase in the AuthenticationProviderManagerTest.

Resolves: FLOW-201

[FEATURE] ArrayMerge supports merging simple types and arrays

ArrayMergeRecursiveCallback is a new method that accepts a closure to map any non array type to an array in a custom way. This is to allow merging in a case where either side is an array and the other not.

Related: NEOS-1004

[FEATURE] Make introduced properties known to persistence

Properties being introduces via AOP are now correctly picked up by Doctrine persistence. To achieve this, the introduced property is made known to the Reflection Class Schema and class properties as if it was a “real” property of the particular class.

Resolves: FLOW-191

[!!!][BUGFIX] Generate Value Object hash from property values

This changes the Value Objects’ hash generation algorithm to use the actual properties of the Value Object, not only the constructor arguments which lead to duplicate hashes in cases where arguments were empty.

Furthermore the initial approach disregarded everything that can be done within the constructor (such as trimming, calculations, etc.).

This patch delays the generation of the value hash to just after instantiation, where the Value Object is completely initialized. The final object properties names and values are then hashed.

Additionally, the handling of DateTime objects has been improved. The generated ValueHash now also includes information about the timezone.

This might be breaking in the unlikely case where the initial hash calculation leads to the same hash as the new calculation, for different VOs. Besides, it might lead to duplicate VOs in the database because the “same” VO can become a new hash due to the changed hashing algorithm.

[TASK] Unclutter Storage API and keep upload internal

This removes the importUploadedResource() method from Collection and WriteableStorage to clean up the interface. Uploaded files are now preprocessed in the ResourceManager and then handled via the importResource() method. Additionally the $filename argument is removed from Collection::importResourceFromContent() and Storage::importResourceFromContent() as it can be handled in the ResourceManager.

Also includes fixes for environments with activated open_basedir directive.

[BUGFIX] Interface object configuration has no effect

This fixes an issue with the object configuration for interfaces which resulted in possibly defined configuration objects in Objects.yaml to be ignored.

Even though Flow internally uses objects in dependency injection referenced through the interface name (for example “PackageManagerInterface”), the object configuration for such objects did not actually work. The error only remained undiscovered because the object configuration of the respective implementation classes had meaningful options set.

With this change applied, it is now possible to safely inject an “interface object”.

Resolves: FLOW-187

[TASK] Fix unit tests failing on PHP 5.6

Some unit tests failed under PHP 5.6 because an argument to be passed by reference was by value. The affected tests now use _callRef in the AccessibleMock.

This should solve those issues on HHVM as well.

Related: FLOW-194

[BUGFIX] PersistentObjectConverter works with “immutable” properties

The PersistentObjectConverter sets convertedChildProperties on the object after it was created (either newly constructed or hydrated from persistance). Creating a new object will filter constructor arguments from the convertedChildProperties but if the object already existed that does not happen. This poses a problem for objects that accept arguments in the constructor that are not settable afterwards. So those properties are considered “immutable”. In cases where you cannot be sure if an object already exists and you give the identity and all properties of an object with “immutable” properties the property mapping will fail if the object already existed as the converter tries to set the “immutable” properties as well.

With this change we check for this kind of properties and compare the given value with the already set value. In case they are identical we ignore the given value and proceed with the property mapping. In case they differ there is an inconsistency in your data that we cannot handle and so throw an exception. This is not breaking as before in all cases an exception would be thrown.

Fixes: NEOS-937

[!!!] Do not use LoggerFactory in a static context

First step to build a more configurable system for factory injection to be used to replace the logger with monolog.

This is breaking in case you rely on the create method of the LoggerFactory being static.

[TASK] Don’t skip core migrations for packages that are not the root of a git repository

This is a follow-up to If66a2dff21b239963728963f15437599a8442f72 that reverts the new behavior of skipping packages that are not the root of a git repository.

Related: FLOW-179

[!!!] Add charset and collation to all MySQL migrations

This change set adds charset and collation to create table statements in the existing migrations. This make sure the tables are set up correctly independent of the database default configuration.

Also migrations generated contain this information since a while, leading to problems on migration if the database is not using the same charset and collation.

This is breaking if you have existing tables that do not use the utf8 charset and utf8_unicode_ci collation. To solve this you need to convert the existing tables. This can be done using the command:

./flow database:setcharset

Related: NEOS-800

[BUGFIX] Account tagging causes Access Denied

This fixes an issue with the account session tagging feature which has been merged minutes ago. Content security blocked the retrieval of the Account object from the content repository. Therefore, the Account methods must be called with temporarily disabled authorisation checks at this stage.

See also change I2ab10b535cea0c80aaff287e65511ea581681379

[FEATURE] Automatically remove sessions of deleted account

This change adds a mechanism which automatically destroys all sessions started by a particular account when that account is going to be removed.

It also introduces a new method “destroySessionsByTag()” to the Session Manager.

Resolves: FLOW-186

[FEATURE] Tag sessions with current account

This change adds an account tag to all sessions started through authentication. Through this tag it is possible to find all sessions of a particular user (ie. account) through the session manager.

Example:

$sessions = $sessionManager->getSessionsByTag('TYPO3-Flow-Security-Account-' . $account->getAccountIdentifier());

Resolves: FLOW-184

[FEATURE] Add command to set charset/collation on MySQL

This adds a new command to set the character set and collation used in MySQL:

./flow database:setcharset

It will convert the database configured in the settings and all tables inside to use a default character set of utf8 and a default collation of utf8_unicode_ci. If needed, those defaults can be overridden.

It will also convert all character type columns to that combination of charset and collation.

Related: NEOS-800

[FEATURE] Use Doctrine ORM 2.4

This change updates the requested Doctrine ORM version from 2.3 to 2.4.

The FlowAnnotationDriver is adjusted to match the features found in the AnnotationDriver of Doctrine ORM 2.4 and is cleaned by importing classes.

One notable addition is the support for the EntityListeners annotation.

Resolves: FLOW-198

[TASK] Removed deprecated use of Inject for settings

This removes the use of the Inject annotation for settings and instead uses the new InjectConfiguration annotation instead.

Related: FLOW-148

[!!!][FEATURE] Make ignoreTags configuration more flexible

This change makes the TYPO3.Flow.reflection.ignoreTags setting a dictionary to allow for adding and changing tag ignore behavior from 3rd party packages.

The previous syntax:

TYPO3:
  Flow:
    reflection:
      ignoredTags: ['tag1', 'tag2']

is now deprecated in favor of:

TYPO3:
  Flow:
    reflection:
      ignoredTags:
        'tag1': TRUE
        'tag2': TRUE

The old syntax is still evaluated so this change is mostly backwards compatible. However it changes the behavior so that configuration is now merged rather than replaced. So this is a breaking change if a package relied on this behavior. To remove a tag from the list of ignored tags, it has to be set to FALSE explicitly now:

TYPO3:
  Flow:
    reflection:
      ignoredTags:
        'someTag': FALSE

Resolves: FLOW-199

[FEATURE] Improve handling of core migrations

Features of this change:

  • “version” flag to allow execution/fetching status of single migrations
  • “verbose” flag to reduce noise if not needed
  • even migrations with no changes are recorded (as empty commits)
  • custom description for migrations (migration class doc comment)
  • skips “TYPO3.*” packages by default (overridden when specifying the –package-key argument)
  • Refactor scripts to ease maintenance

Resolves: FLOW-179

[!!!] Exclude Non-Flow packages from object management by default

With this change all packages, that are not of one of the “typo3-flow-*” composer types, are excluded from object management by default.

Previously the had to be excluded explicitly with the TYPO3.Flow.object.includeClasses setting.

To activate object management for Non-Flow packages, the newly introduced setting TYPO3.Flow.object.includeClasses can be used. It works in the same way as excludeClasses, apart from not allowing wildcards for the package.

This is a breaking change in case proxy building for non-flow packages was expected. In these cases packages have to be included explicitly now:

TYPO3:
  Flow:
    object:
      includeClasses:
        'non.flow.package' : ['.*']

To exclude classes from Flow packages a non-matching or empty expression can be specified:

TYPO3:
  Flow:
    object:
      includeClasses:
        'Some.Flow.Package' : []

The excludeClasses setting is deprecated but still evaluated.

Resolves: FLOW-103

[FEATURE] The Query Object Model supports distinct queries

This changeset introduces the methods getDistinct and setDistinct on the Query object to allow queries to explicitly return only distinct result sets, which might be needed for join queries which happen implicitly in subproperty queries, e.g. property.subProperty.foo IN (1,2,3)

Doctrine automatically hydrates only distinct entities on result sets, but that happens only after a limit clause on the query. This leads to wrong query results with limit clauses, with less entities than distinct existing entities in the database. A test is provided that shows the behaviour.

Resolves: FLOW-21

[TASK] Explain type attribute for InjectConfiguration annotation

This fills a small gap in the documentation for the new configuration injection mechanism.

Related: FLOW-148

[FEATURE] Support for variables in routing default values

Currently placeholders are only supported in name and uriPattern.

This adds support for default values allowing for better reusability of similar routes. For example a main Routes.yaml with:

-
  name: 'CRUD - product'
  uriPattern: '<CRUDSubroutes>'
  subRoutes:
    'CRUDSubroutes':
      package: 'Acme.Package'
      suffix:  'Crud'
      variables:
        'resourceName': 'product'

And the corresponding sub routes Routes.Crud.yaml with:

-
  name: '<resourceName> - index'
  uriPattern: '<resourceName>s'
  defaults:
    '@controller': '<resourceName>'
    '@action': 'index'
  httpMethods: ['GET']

-
  name: '<resourceName> - create'
  uriPattern: '<resourceName>s'
  defaults:
    '@controller': '<resourceName>'
    '@action': 'create'
  httpMethods: ['POST']

-
  name: '<resourceName> - show'
  uriPattern: '<resourceName>s/{<resourceName>}'
  defaults:
    '@controller': '<resourceName>'
    '@action': 'show'
  httpMethods: ['GET']

Resolves: FLOW-76

[FEATURE] JsonView accepts encoding options

json_encode supports multiple bitmask options. see: http://www.php.net/manual/en/json.constants.php

These options are supported with this commit using the supported options of AbstractView

Usage Example: $this->view->setOption(‘jsonEncodingOptions’, JSON_FORCE_OBJECT | JSON_NUMERIC_CHECK);

Resolves: FLOW-157

[FEATURE] Embedded Development Web Server

By using ./flow server:run, a quick-and-dirty development server is started; so no web server configuration is needed anymore.

Resolves: FLOW-169

[FEATURE] Allow custom handling of propertyMapping errors

Previously if an entity wasn’t found during property mapping an exception was thrown before the action was invoked making it difficult to change the default behavior of showing a 404 error.

This change adjusts the PersistentObjectConverter to return a TargetNotFoundError in that case instead of throwing an exception. It also extends the \\TYPO3\\Flow\\Error\\Result class by a method getFlattenedErrorsOfType() that allows to retrieve all errors implementing a given class or interface.

The default errorAction of the ActionController now checks the validation result for TargetNotFoundErrors and throws an exception if that’s the case. But this behavior can now be changed by overriding handleTargetNotFoundError():

protected function handleTargetNotFoundError() {
  try {
    parent::handleTargetNotFoundError();
  } catch (TargetNotFoundException $exception) {
    // custom behavior (e.g. redirect to some action)
  }
}

Resolves: FLOW-197

[FEATURE] ObjectConfiguration gets name from annotation

Currently, an object configuration’s property’s class must explicitly configured with its name, even if the name is implied by the annotation of the intended property.

This change falls back to that annotated class name, if the name is not explicitly mentioned. For example, this configuration now will work:

'Acme\\Acme\\SomeClass':
  properties:
    'someProperty':
      object:
         # the type of 'someProperty' will be inferred from the var annotation now
         # previously it had to be specified via `name`
        arguments:
          1:
            value: 'SomeConstructorArgument'

…as long as the mentioned property someProperty has a proper @var annotation revealing the class name.

[BUGFIX] Reset SecurityContextHash on logout

Adjusts AuthenticationProviderManager::logout() to reset the ContextHash of the Security\\Context whenever an account is logged out in order to prevent invalid caching entries.

Related: NEOS-433

[TASK] Respect “SecurityContextHash” in doctrine caches

Adjusts the Doctrine\\CacheAdapter to include the current SecurityContextHash whenever writing cache entries in order to prevent protected entities to be available to unauthorized users.

Related: NEOS-433

[BUGFIX] Start session when fetching a CSRF token

This change adds a @Flow\\Session(autoStart=true) annotation to the method Security\\Context::getCsrfProtectionToken().

Background:

Currently CSRF tokens are bound to a session. Thus fetching a token without starting a session makes no sense because the token will be invalid on the next request.

In the long run we might be able to create “stateless” CSRF tokens that don’t require a session.

Related: FLOW-130

Depends: I896f6a722445deede1f0a656ea73db04f0d2e978

[BUGFIX] Enforce CSRF token for sub requests

With this change dispatching of requests is intercepted recursively so that a valid CSRF token is enforced for sub requests, too. Previously the token was only enforced on the main ActionRequest.

Background:

Previously the CSRF token was enforced via an AOP aspect. But one aspect can only be executed once at a time. So calls of Dispatcher::dispatch() that are invoked during the execution of the same method (which is the case for plugin or widget sub requests) weren’t intercepted by the aspect.

This change removes the aspect in favor of a hard coded check in the Dispatcher class.

Related: FLOW-130

[FEATURE] Add HTTP-version and start-line support to Http\Message

This adds support for the HTTP-version to HTTP Messages like Request and Response; i.e. it is stored and accessible.

As a consequence, convenient getter methods for the so-called “Start-Line” (RFC 2616, section 4 HTTP Message) is added, being either the “Request-Line” or the “Status-Line” depending on the Message implementation (being a Request or a Response). See RFC 2616, sections 5.1 and 6.1 accordingly.

Besides, it makes Http\Message an abstract class because an HTTP message must be a Request or a Response and cannot be a Message itself.

[TASK] Embedded Entities in ObjectArray are lazy loaded

The ObjectArray would fetch doctrine entities one by one at the time the main entity was hydrated, now we create a lazy loading proxy instead. This should have no negative effects, if all embedded objects are used the same amount of queries will happen, but if none of the embedded objects is needed, then they won’t be fetched from persistence at all.

[BUGFIX] Fix typo in PHPDoc for method getParameters

This commit fixes a typo in the PHPDoc for the method getParameters in file TYPO3\Flow\Reflection\MethodReflection

[BUGFIX] Fix typo in PHPDoc for method createSchema

This commit fixes a typo in the PHPDoc for the method createSchema in file TYPO3\Flow\Persistence\Doctrine\Service

[TASK] Suggest ext-curl in composer manifest

The curl PHP extension is used in the HTTP client CurlEngine and thus this change marks it as suggested in the composer manifest.

[TASK] Remove use of deprecated getResourcePointer

The ArrayConverter still referenced the deprecated method getResourcePointer() in the code used to export file data contained in Resource objects.

[!!!] Remove obsolete “security.enable” Setting

This change removes the TYPO3.Flow.security.enable and all mentions and usages of it.

Background:

This setting was initially intended for performance reasons (For applications without security features) and in order to disable security for (functional) tests. For the latter we use a different approach since a while and the performance hit of security features is also neglect-able since Flow pre-compiles classes (at least if there is no complex policy configured). Besides the flag was never evaluated consistently.

Resolves: FLOW-181 Related: FLOW-11

[BUGFIX] Properly support doctrine’s indexBy attribute

Doctrine allows an indexBy attribute at OneToMany and ManyToMany relations. The current FlowAnnotationDriver removes this attribute. This patch passes it forward again.

Functional tests are included.

[TASK] Use UTF-8 safe parse_url in Flow

This adds parse_url() to the Unicode\Functions class and makes use of it throughout Flow.

See https://bugs.php.net/52923 for some background.

[TASK] Make i18n locale fallback rule handling a bit more robust

If the locale fallback rule for the i18n framework is given without order an exception is thrown. A missing strict flag on the fallback rule is set to the default (FALSE) as implied by the documentation.

[TASK] Remove deprecated classes and methods

This removes everything marked deprecated in 2.0 and before. Actually we should also remove everything that was deprecated from 2.1 but as we were so lenient with the 2.0 things, I left that alone for now.

[BUGFIX] Constraint with “IN” and empty collection should work

Entity constraints using “IN” where the argument resulted in an empty array would generate a query that contained an empty IN() operation. This breaks at least in MySQL. The only way to test for NULL is a IS NULL constraint. This change takes care of that. The changed test exposed the issue.

[TASK] Ignore unknown Roles in Account->hasRole()

As a followup to the change I10968698163d70b9ea387b098eb3bb46ed09c98f this addresses the concern about hasRole() being inconsistent now.

[!!!][BUGFIX] SessionManagerInterface and SessionInterface are incomplete

This change adds functions which have been around for some time now in Session and SessionManager to their respective interfaces.

TransientSession now also implements these previously missing methods.

This patch is breaking in the unlikely case that you implemented your own Session or SessionManager implementation and forgot to implement the methods mentioned in the classes but not yet mentioned in the interfaces.

[TASK] Ignore invalid roles in Account->getRoles()

This change adds a safeguard which ignores role identifiers which are possibly still stored with an account, but refer to roles which do not exist anymore, or at the moment. Previously Account->getRoles() would throw an exception when it stumbled over a non existing role.

[TASK] Fix warnings during reStructuredText rendering

This tweaks rst files to get rid of some warnings that are emitted during documentation rendering.

[TASK] Remove leftover table

This change removes the typo3_flow_resource_publishing_abstractpublishingconfiguration table that should not be present.

Resolves: FLOW-185

[TASK] Improve CSRF log entries

This change tweaks the log/exception messages of the CsrfProtection RequestPattern.

It also adjusts the behavior to log if

  • CSRF enforcement was skipped due to a “skipcsrfprotection” annotation
  • CSRF token was successfully verified

Related: FLOW-130

[TASK] Remove inaccessible code from ArrayConverter

Remove code that is currently inaccessible, because the constant STRING_FORMAT_SERIALIZED is not defined in the class.

Unserializing from untrusted sources should not be done anyway so we remove this possibility completely instead of adding the constant.

A use case which would require an unserialize for array conversion is complex enough to be handled in a dedicated type converter class which exactly fits the use case instead of providing a potentially insecure shortcut for that (unserialize) in the framework.

[TASK] Adjust Policy schema to new format

This is a follow-up to the “Restructure policy component to new Policy.yaml format” change (I84e188e89a05ec0dd1f9ee96fe312dac81806759) adjusting the schema according to the new syntax.

Related: FLOW-11

[TASK] Introduce privilege subjects

This change introduces a privilege subject interface and a method implementation to pass method invocations as subject to the method privilege implementation.

Related: FLOW-11

[TASK] Tweak error handling in doctrine:migrationversion

Instead of an uncaught exception, two expected error states are now handled in a more friendly way.

[BUGFIX] Adjust settings schema to new Resource Management

This fixes the TYPO3.Flow.persistence.schema.yaml according to the “Multi-Storage / Multi-Target Resource Management” feature introduced with Ia2b47b4070a2dfabf4833bf1f0f3967ba3b032a7.

Besides this removes an obsolete “detectPackageResourceChanges” setting.

Fixes: FLOW-129

[BUGFIX] Adjust settings schema to “Add Configuration for Doctrine Filters”

This fixes the TYPO3.Flow.persistence.schema.yaml according to the “Add Configuration for Doctrine Filters” change introduced with If8582f8d138a7e46b8b77fc3c4b83b78bfc93bba.

[!!!][BUGFIX] Correct object modification exception trigger

The PersistentObjectConverter throws an exception if there are properties to be set on the object and modification was not allowed in the PropertyMappingConfiguration. The decision if there are properties to be set was done based on the amount of entries in the $source array, but in fact only the $convertedChildProperties are set to the model, so the check should check if there is anything in that array.

That means you can have any amount of arbitrary data in your data source as long as it is not converted to an actual child property. Which is determined by the getSourceChildPropertiesToBeConverted method of the converter.

This is breaking if you rely on the fact that the exception is thrown if you have arbitrary data in $source even though that data would never have been set to your model.

[BUGFIX] Package meta data do not contain package type.

When loading packages using the PackageManager class, the associated MetaData instance is not initialized with the package type.

[TASK] Use unicode-safe pathinfo function and use it where necessary

pathinfo() function is not unicode-friendly if setlocale is not set. It’s sufficient to set it to any UTF-8 locale to correctly handle unicode strings. This change temporarily sets locale to ‘en_US.UTF-8’ and then restores original locale. It’s not necessary to use this function in cases, where only file extension is determined, as it’s hard to imagine a unicode file extension.

Related: FLOW-101

[BUGFIX] Respect correct property filling priority in ObjectConverter

The order preference to try to set a property via constructor, via setter and via public property is now respected correctly by the ObjectConverter::getTypeOfChildProperty() method. That method used to check the setter annotation in the first place, then the constructor annotation in the second place, and failed when a property was only settable through its public nature since that case was not backed at all.

The checking/setting priority now follows the one used by ObjectAccess, so it is Constructor > Setter > Field.

Fixes: FLOW-33

[!!!][FEATURE] Introduce InjectConfiguration Annotation

This adds a new InjectConfiguration annotation that can be used to easily inject settings or other configuration types to classes.

Example:

/**
 * @var string
 * @Flow\\InjectConfiguration("my.setting")
 */
 protected $mySetting;

/**
 * @var string
 * @Flow\\InjectConfiguration(package="TYPO3.Flow", path="core.phpBinaryPathAndFilename")
 */
protected $phpBinary;

/**
 * @var array
 * @Flow\\InjectConfiguration(type="Views")
 */
protected $viewsConfiguration;

This change is marked breaking because it deprecates setting injection via the Inject annotation (introduced with Id84d087307d348ecd3079fc6097df193ebecb08a).

It also reverts support for the InjectSettings annotation that has been introduced with Iaec291e40ffd352de9810c4e72027c455bf8c566 (but was never part of a release).

Related: FLOW-148

[BUGFIX] AOP works with __clone call on parent objects

If, for example, you extend an entity which implements __clone the AOP Framework breaks with an warning in development mode, that it cannot access Flow_Aop_Proxy_targetMethodsAndGroupedAdvices.

Solution is to check if the private Flow_Aop_Proxy_targetMethodsAndGroupedAdvices property is accessible and otherwise skipping the Advice call.

[BUGFIX] Log exceptions recursively in SystemLogger::logException()

With this change “post mortem” information about the complete exception chain is logged, and not only for the outer exception.

This also adds some cosmetic and non-functional cleanups in order to increase readability and IDE support:

  • Import FQN where applicable
  • Remove unused import statements

Fixes: FLOW-159

[BUGFIX] Adjust CommandLine documentation to refactored console output

This adjusts the CommandLine section of the documentation to changes introduced with Ia77c62b41fb598bdfb7b81c530494ba819a590d1.

[TASK] Update documentation for the new resource management

Resolves: FLOW-114

[BUGFIX] Adjust settings schema to “Add a generic lock class”

This fixes the TYPO3.Flow.utility.schema.yaml according to the “Add a generic lock class” change introduced with Ib5cacb4e8a0784814bd863ae19b591acd540e4ef

Besides this puts the lockStrategyClassName setting in quotes as suggested in the original change.

[BUGFIX] Disable security for CLI requests

Currently it’s not possible to invoke methods that are covered by a policy via CLI because the security context is not yet initialized leading to a The security Context cannot be initialized yet exception.

With this change all authorization checks are disabled for command controllers.

Fixes: #FLOW-163

[TASK] Cosmetic cleanup in CLI and bootstrap classes

This is a non-functional change that incorporates following adjustments in order to increase readability and IDE support:

  • Replace magic strings “Runtime” and “Compiletime” by constants
  • Import FQN where applicable
  • Remove redundant doc comments
  • Inline @var annotations where applicable
  • Commit: 294a2a6

[FEATURE] Automatically move generated migration to package

This adds some interaction to the doctrine:migrationgenerate command allowing to move generated doctrine migrations to the specified package.

Example output:

Do you want to move the migration to one of these Packages?
  [0 ] Don't Move
  [1 ] TYPO3.Fluid
  [2 ] TYPO3.Eel
  [3 ] TYPO3.Flow
  [4 ] TYPO3.Party
  ...

[TASK] Cleanup PersistentObjectConverter and tweak InvalidSourceException

This is basically a cosmetic change to the PersistentObjectConverter and corresponding unit test which incorporates following non-functional changes:

  • Import FQN for better readability
  • Adjust @throws, @param and @return annotations for better IDE support

Additionally this adjusts the exception message of the InvalidSourceException in fetchObjectFromPersistence() to prevent fatal errors when the identity is of an invalid type and adds a corresponding test.

[BUGFIX] Fix duplicate keys in Testing/Settings.yaml

[BUGFIX] Package keys with different case should not be allowed

Composer packages could change their Flow package key case, the package manager needs to prevent registering the same package twice.

Fixes: FLOW-156

[BUGFIX] Properly resolve case of Subpackage Key in ActionRequest

ActionRequest::getControllerSubpackageKey() failed to return the correctly cased subpackage key. This is not the case for the other getController*() getters and can lead to issues (e.g. “Template could not be loaded” Fluid exceptions on case-sensitive file systems.

This change adjusts the getControllerSubpackageKey() method to getControllerName() which already uses the correctly cased controllerObjectName to extract the controller name.

Fixes: FLOW-126

[BUGFIX] Documentation: Correct pagination widget example

The example code for a fluid widgets uses a not working syntax of the pagination widget.

Move the configuration for “itemsPerPage” into the correct attribute.

Fixes: FLOW-100

[FEATURE] Filesize utility functions

This adds two new convenient functions to Utility\\Files that allow for converting a number of bytes to a human-readable representation vice versa.

Usage:

\\TYPO3\\Flow\\Utility\\Files::bytesToSizeString(1073741823);
\\TYPO3\\Flow\\Utility\\Files::sizeStringToBytes('1024M');

Related: NEOS-842

[FEATURE] Allow to send custom request headers automatically

The Browser provides a method to set headers to be sent with every request now:

$browser->addAutomaticRequestHeader('Accept-Language', 'lv');

Removal of a previously added header is possible with:

$browser->removeAutomaticRequestHeader('Accept-Language');

[BUGFIX] Get rid of TYPO3CR dependency in integration tests

This change extracts the trait inclusion in a package specific behat helper class and not directly into the command controller. By this, every package can provide the traits needed by its tests based on this helper class.

Fixes: FLOW-134

[FEATURE] Add PackageKeys as namespaces to TemplateParser

This change registers a Fluid ViewHelper namespace for every active package.

This means, that you can call any package viewHelper without declaring a namespace like this:

<acme.somepackage:someViewHelper />

Depends: Ie4e40713ec7b2a31464ddd633458d757d55d52e7

Related: FLOW-151

[!!!] Introduce InjectSettings Annotation

This change mainly cleans up injection code by moving reading of injection annotations to the ConfigurationBuilder from the ProxyClassBuilder, so that the ProxyClassBuilder again mostly works based off the given configuration.

Additionally property injection was moved to a separate annotation that now allows injection of whole package settings from a separate package with the following syntax:

@Flow\\InjectSettings(package="TYPO3.Party")

will inject all settings for the package TYPO3.Party.

Just using the annotation like this:

@Flow\\InjectSettings

will inject all settings for the package in which the class with the annotation is in.

Giving a specific setting path is also possible, with or without the package. So this:

@Flow\\InjectSettings(package="TYPO3.Flow", path="i18n.defaultLocale")

Will inject the Setting TYPO3.Flow.i18n.defaultLocale regardless in which class the annotation was used.

This change is marked breaking as injection of settings via the Inject annotation is from now on deprecated and will be removed in three versions.

Resolves: FLOW-148

[TASK] Tweak Quickstart tutorial

Adjusts the Quickstart to be in sync with recent Flow changes.

Related: FLOW-139

[BUGFIX] Throw exception when trying to reflect a non-existing class

Previously the ReflectionService ignored classes that couldn’t be loaded. It just logged an error and marked the respective class “unconfigurable”. This leads to weird side effects that are hard to track down.

With this change an exception is thrown during compile time whenever a class is being reflected that couldn’t be loaded.

If a class or file should be skipped during reflection, the excludeClasses can be used:

TYPO3:
  Flow:
    object:
      excludeClasses:
        'Some.PackageKey': ['Some\\\\Class\\\\Name']

Fixes: FLOW-128

[TASK] Explain “inconsistent naming” of classes and interfaces

This adds the wonderful explanation the reasons for our naming of classes and interfaces that Jacob Floyd sent to the mailing list to the CGL appendix.

[BUGFIX] Adjust settings schema to “HTTP components for handling requests”

This fixes the TYPO3.Flow.http.schema.yaml according to the “HTTP Components” feature introduced with I1e2491dba5adc125a7b85a574c9b51c9ae2ff18f

Fixes: FLOW-35 * Related: #52064 * Commit: b6e8816

[BUGFIX] PHP Notice when multiple namespace roots are set

The Package class triggers a PHP Notice when a composer manifest contains several search paths for the same prefix (see [1]):

{
  "autoload": {
    "psr-0": {
      "Foo\\\\": ["src/", "tests/"]
    }
  }
}

This commit changes the behaviour to using the first path as class path when multiple paths are defined.

[1] https://getcomposer.org/doc/04-schema.md#psr-0

Resolves: FLOW-94

[BUGFIX] Make sure functional tests have default resource setup

Duplicates the default resource configuration to the testing context to make sure that functional tests have a default environment to run in. Additionally sets the publishing target to a special testing path to avoid problems of resource removal.

[BUGFIX] Support custom factories for constructor argument injection

It should be possible to use custom factories (factoryObjectName) for constructor injection (arguments) the same way as they can be used for property injection.

Resolves: FLOW-135

[TASK] Array converter should use streams to copy resource files

[FEATURE] Collection and object to array converters

Adds two TypeConverters to convert objects to arrays and Doctrine Collections to arrays.

[TASK] Adjust documentation to PHP 5.5

Adjust version requirements and remove magic quotes hint.

Related: FLOW-124

[BUGFIX] Use bin2hex in ObjectArray when using PostgreSQL

The ObjectArray type still uses serialize() to convert to the database value, thus producing data that cannot be used as is for a BYTEA column.

With this change, the serialized string is run through bin2hex() when writing to the database and through hex2bin on the way back, if the system runs on PostgreSQL.

Fixes: FLOW-132

[FEATURE] Command for detecting and cleaning up broken resources

This introduces a new command “resource:clean” which allows for detecting resources which have no corresponding data anymore. It also resolves related Asset objects from the TYPO3.Media package (if installed) and, if requested, removes all broken resources including their assets from the database.

Resolves: FLOW-131

[TASK] Throw more meaningful exception if resource could not be published

This throws a more meaningful exception if a resource as part of a collection to be published had no accesible data (no source stream).

[FEATURE] ResourceTypeConverter allows setting of collection

Adds PropertyMappingConfiguration options for the ResourceTypeConverter to directly set the used collection via CONFIGURATION_COLLECTION_NAME or alternatively set it via __collectionName in the $source.

Resolves: NEOS-787

[BUGFIX] Regression in core migration 20141113121400

In change I30de07c0bb5d322f1b8aa64d1cc890ebbe4c9ab9 we modified the core migration Version20141113121400. However, “$this” is still not allowed in a use() statement, also not in PHP 5.5.

Instead, $this is available in anonymous functions without any use statement.

Related: FLOW-124

[BUGFIX] “Session Not Started” exception

The sole existence of the method Resource->__destruct() leads Doctrine to proxy that method and run __load() before __destruct(), which in turn will triger the SQL protection in Flow Security, which will then discover that a possibly previously existing session has been half-destroyed already.

So we go the safe way and use Flow’s shutdown mechanism instead.

Resolves: FLOW-121

[TASK] Better way to close resource source stream

Using RackspaceCloudFiles I experienced “too many open files” errors. This led me to change the point where one should close a resource stream: instead of doing it in publishFile() I put the fclose() closer to the getStream() call so you can actually see when the stream has been fetched and when it has been closed.

Resolves: FLOW-122

[FEATURE] Pointcut constraints on annotation properties

With this change it is possible to add constraints on annotation property values to the classAnnotatedWith and methodAnnotatedWith pointcut filters.

The following notation becomes valid then:

methodAnnotatedWith(TYPO3\\Flow\\Annotations\\Session(autoStart == TRUE))

[BUGFIX] Prevent iteration over empty collection

This shouldn’t make a difference but with the current way properties are serialized in the TYPO3CR it can happen that you have ArrayCollections which do contain a NULL value instead of an array. This will break on the next serialization and this prevents it.

[TASK] Remove unused flag from BaseTestCase

That flag has been deprecated sine PHPUnit 3.3, so it is time to say goodbye.

[FEATURE] Allow privilege evaluation for arbitrary roles

This adds two methods isGrantedForRoles() and isPrivilegeTargetGrantedForRoles() to the PrivilegeManagerInterface and its default implementation.

This allows to test privileges for roles independently from the currently authenticated account.

Related: FLOW-11

[FEATURE] Respect implementation of JsonSerializable interface

The JsonView will call jsonSerialize() in transformValue() for objects implementing the JsonSerializable interface. This is useful if a domain model or data transfer object needs custom serialization logic for JSON.

[TASK] Fix duplicate exception codes

These exception codes were copied from the Eel package, now they are timestamps fresh off the press.

[TASK] Add “suggest” and “conflict” dependencies to newly created composer manifests

Newly created composer manifests only contained the “require” dependencies of the package meta data. With this change also suggested and conflicting dependencies are added.

Related: NEOS-785

[TASK] Adjust to PHP 5.5 requirement, remove checks, fix date.timezone

The constant for the minimum PHP version has been raised to PHP 5.5.0 and some code that existed purely for backwards compatibility with older PHP versions has been removed.

This change removes a few checks for installed PHP extensions or PHP maximum versions which are still from the PHP 6 era and are no longer necessary. It also removes the dependency on ext-session (since we don’t use it anyway) but declares the dependency to ext-mbstring (since we do use that one).

We also don’t set unicode related ini values which were only supported by PHP 6. The check for the magic quotes setting is also now gone.

And finally, we don’t require date.timezone to be set. Still, PHP does require date.timezone to be set as soon as you are using date functions. In order to still have a smooth setup experience we turn a blind eye on this setting and simply configure the timezone to UTC if it hasn’t been configured by the lazy server admin.

Resolves: FLOW-124

[TASK] Make “renderingGroup” available to custom exception handlers

This change adjusts the exception handling slightly to make the resolved “renderingGroup” available to custom exception handlers.

Besides, this sets the Fluid StandaloneView request package to “TYPO3.Flow” for depending ViewHelpers to work properly.

This is currently required for TYPO3.Neos in order to localize the error messages.

Related: NEOS-497

[TASK] Add missing doc comments in ResourceManager

Resolves: NEOS-789

[BUGFIX] Fix postPackageUpdateAndInstall() in Flow

When a package declares it has some resource to install by defining:

"extra": {
    "typo3/flow": {
        "manage-resources" : true
    }
}

in the composer manifest, the contents of the Defaults and Essentials folders in Resources/Private/Installer is supposed to be copied to the project root.

This was broken, the files were copied to their own source location instead, breaking changes like https://review.typo3.org/34312

Fixes: FLOW-120

[BUGFIX] Fix getStaticResourcesWebBaseUri()

This fixes the deprecated method getStaticResourcesWebBaseUri() in the ResourcePublisher and adds some logging for usage of the deprecated methods.

Resolves: FLOW-118

[TASK] Fix a doc comment in ResourcePublisher

[FEATURE] Add entity privilege target for Doctrine persistence

Adds a new privilege type, beeing able to filter all Doctrine queries for entities the current roles should not be allowed to see.

This is working for all entities retrieved from persistence via Doctrine, no matter if it’s done with DQL, QOM or while lazy loading relations.

Resolves: FLOW-10

[TASK] Move privilege evaluation into privilege manager

To avoid usage of a static vote functions in privilege classes, this change moves evaluation of privileges into the privilege manager. This change removes the concept of privilege voters, which is not needed due to the posssibility of implementing custom privilege types.

This change also fixes an inconsistency within the privilege evaluation process: Privilege targets with runtime evaluations will no longer taken into account, if the runtime constraint does not match the current situation.

When setting the same privilege twice within the same role, only the last one will have effect. With that it is actually possible to override permissions, e.g. in a dependant package.

Related: FLOW-11

[!!!][FEATURE] Multi-Storage / Multi-Target Resource Management

This change introduces a revised resource management which allows for storage and publication of persistent or static resources (assets) in the local file system or other services, such as Amazon S3 or Rackspace CloudFiles. It also introduces the concept of collections which allows for grouping resources into collections with specific storage and publication rules.

Existing persistent resources are migrated through the Doctrine migration contained in this feature.

Note: this change raises the PHP requirement to 5.5.

Resolves: FLOW-108

[BUGFIX] Memcached backend not cleared across Cli/Web requests

This is because the backend is bound to the executing script + SAPI mode and not the installation path + Flow context. This makes it impossible to clear entries created in the Web with the Cli cache flush commands, and vice versa.

Additionally the Flow context is not taking into account so the Development/Production share the same cache, which can lead to undesired behavior.

Fixes: FLOW-116

[TASK] Remove unused test fixture

[!!!][BUGFIX] Skip automatic persistence for updated entities

When trying to persist changes in a “safe request” (e.g. GET) Flow throws an exception:

Detected modified or new objects [...] to be persisted which is not
allowed for "safe requests"

including details on how to work around this.

This currently only works if entities have been added or removed.

With this change also updates to entities are tracked correctly so that automatic persistence is skipped for modified objects for safe requests.

This is a breaking change when code relied on the incorrect behavior of automatically persisting changes even for safe requests. In this case make sure to trigger updates only via unsafe requests (e.g. POST or PUT). If that’s not an option, the issue can be worked around with a manual call to PersistenceManager::persistAll().

Fixes: FLOW-84 * Related: #47252 * Related: #51570

[TASK] Add changelog for TYPO3 Flow 2.3.0-beta1

This adds the 2.3.0-beta1 change log to the master branch.

See https://ci.neos.typo3.org/job/typo3-flow-release/23/

[BUGFIX] Mark security tests using static mocks incomplete

This marks five tests that use static mocks as incomplete. Static method mocking is no longer supported since PHPUnit 4.0, and the test failures do not indicate a broken functionality.

Instead they might mask other test failures, because “everyone knows” the build is broken by these tests...

[!!!][FEATURE] Restructure policy component to new Policy.yaml format

This change introduces the new concept for policies and privileges. It also includes a restructuring of the privilege voting process.

This is a breaking change mainly because it drops support for content security and secure downloads. Both features will be re-added by new privilege types in separate changes. Besides it is quite likely that custom code that interacts with the (non-public) API of the security framework won’t work without adjustments.

The new Policy.yaml syntax is covered by code migrations, so make sure to run:

./flow core:migrate
./flow doctrine:migrate

commands and to carefully read their output.

Resolves: FLOW-11

[BUGFIX] Adjust code migration identifier pattern to contain the full timestamp

Previously code migrations are expected to have a class name with the pattern Version<YYYYMMDDhhmm> and the unique identifier was determined extracting the last 12 characters of the class name (which are expected to be the timestamp).

With this change everything after the “Version” string is considered for the identifier, allowing the timestamp to contain seconds as well.

This also adjusts existing code migrations to use the full timestamp in order to establish the new guideline (note: those migrations still return the old identifier so that they won’t be applied again with a new identifier).

Fixes: FLOW-110

[BUGFIX] Make rewriteFilenameForUri handle non-ASCII names correctly

When the filename consists completely of non-ASCII characters, the rewriteFilenameForUri would substitute it with empty filename like ”.jpg”, resulting in a broken resource link.

This change makes rewriteFilenameForUri accept unicode character. In addition it checks if the filename is empty after the rewrite and names it “unnamed.<fileExtension>” if needed.

Fixes: FLOW-99

[TASK] Fix wrong path in documentation

Resolves: FLOW-91

[TASK] Add helper to get the simple type or className of a value

[FEATURE] UriTemplate implementation

This adds UriTemplate as per RFC 6570, allowing expansion of templates into URI strings:

// results in "foo/bar/baz"
UriTemplate::expand('foo/{var}/baz', array('var' => 'bar'));

See the unit test and/or RFC for the long list of possible expansions.

[FEATURE] Allow setting ini entries to sub requests

This adds a configuration option core.subRequestIniEntries where additional INI entries which should be passed to the Flow CLI sub request can be stated.

This is exactly the behaviour of passing such a value via the -d parameter of the php CLI.

[FEATURE] A Translation EelHelper

This adds an EelHelper for fetching translation IDs.

Usage example:

${Flow.I18n.Translation.translateById('someId', 'Acme.Shop')}

[TASK] In Debugger, blacklist Repository and Service objects

In order to reduce the vulnerability of unwanted recursion, properties of objects ending with *Service or *Repository are not rendered again.


This change adds a separate file to track the status of the site lock in addition to the actual lock file.

Besides this registers LockManager::unlockSite() as shutdown function to make sure that the lock is released when the scripts ends, to prevent that a site stays locked if an PHP error or exception occurred.

Background:

Due to a regression introduced with Ib701d67b3c7ed81a09e340583ffd2ee768857b83 the site lock could not be released reliably on Windows systems because the lock file was not unlinked. This is solved by using an additional file to track the status of the lock.

[TASK] Update Release Notes and Upgrading Instructions

This tweaks and finalizes the Flow 3.0 Release Notes and adds an Upgrade Instructions section.

Besides this removes some left-over occurrences of “TYPO3 Flow”.

[BUGFIX] Skip existing files when importing temporary resources

When a file is imported to the WriteableFilesystemStorage it would previously always override the final file even if it already existed. In case this same file has an open file handle, that can fail depending on the used environment (Windows). Cases where the target file exists and is opened can happen easily if a stream was opened to the target file via the ResourceManager. For example if you want to copy an existing Resource this will happen.

[TASK] Remove AvailableProxyClasses when flushing all caches

When caches are flushed via \\TYPO3\\Flow\\Cache\\CacheManager::flushCaches() then the AvailableProxyClasses were not removed. This could lead to weired issues because new classes were not loaded from the proxy cache.

Related: FLOW-291

[BUGFIX] Flush configuration cache upon PackageStates update

This change makes sure that the configuration cache is flushed whenever the PackageStates.php file is being updated.

Background: Since I1f9d560303758c87b81a8b5578f1b80b76deb398 a compile time configuration cache is built even in Development context. When this cache is not flushed after creating a new package, this leads to an fatal error:

Argument 1 passed to ActionController::injectSettings() must be
of the type array, null given

The reason is, that we currently initialize an empty setting for every package in the ConfigurationManager. If the cache is not rebuilt upon creation of new Packages, this initialization is skipped and:

ConfigurationManager::getConfiguration('Settings', 'New.Package');

returns NULL.

Fixes: FLOW-291

[TASK] Update contributors list in guide

[BUGFIX] Avoid “too many open files” exception during I18n filesystem scan

The I18n Service scans the filesystem for localized files in order to build a list of available locales, this is done with a RecursiveIterator, which will keep handles open in nested structures. A simple stack like now used in the FileMonitor prevents the problem of “too many open files” (leading to a fatal error) and at the same time brings clear performance boost. Additionally hidden directories and files (starting with ”.”) are ignored now.

Fixes: NEOS-111 (cherry picked from commit e6b5455225cf649de9ca2a2f919094b07429da01)

[TASK] Add request method in CurlEngineException

This change display the current request method in the CurlEngineException message.

(cherry picked from commit 93a763d6c5a8b60ca74b1640230832ff1d25efb6)

[BUGFIX] Fix configuration option name for token in documentation

The security documentation referred to an authentication provider option tokenClass but the correct setting is token.

Fixes: FLOW-51

[BUGFIX] Improve resolution of core migration version argument

By specifying the version flag on the ./flow core:migrate command a specific core migration can be targeted.

This change improves the regular expression that extracts the numeric identifier from that flag: Previously when specifying the whole version string as in TYPO3.Fluid-20150214130800 the resulting version number would include the “3” of the package key.

Related: FLOW-179

[BUGFIX] Less verbose exception rendering in CLI

This change introduces several improvements addressing rendering of exceptions in CLI mode:

  • Move echoExceptionCli() up to AbstractExceptionHandler to unify rendering throughout different contexts
  • Remove rendering of debug backtrace (this has been introduced with Ic4801947e60738f2693fb75590b5c9ba8b67f0ed but in the meantime the backtrace of Flow exceptions is stored in the log anyways)
  • Use sparse highlighting if available (requires posix PHP extension)
  • Output relative paths not only for files in /Packages

Fixes: FLOW-285

[BUGFIX] Rename resource:repair to resource:clean in the exception message

[TASK] Get package version from composer.lock

Currently the PackageManager don’t show the package version information.

This patch parse the composer.lock to get the current package version. This patch also add internal caching to avoid multiple parsing of the composer.lock

Resolves: NEOS-189 Resolves: FLOW-19

[BUGFIX] Isolated behat steps should clear configuration caches

The change I65211ddd90484ec1a9d970394f0b9d268805ea9f uncovered an issue with isolated behat steps as they relied on the wrong behavior that configuration caches would be cleared on every development request. This change adds clearing of Configuration caches to the other cache clearings happening for each isolated step.

[!!!][TASK] Remove unnecessary use of ReflectionService

This removes a few usages of the ReflectionService and additionally removes initialization from two methods in the ReflectionService as those do not use any state in the ReflectionService.

This change is not breaking in itself but deprecates the method \\TYPO3\\Flow\\Reflection\\ReflectionService::getClassNameByObject() please use \\TYPO3\\Flow\\Utility\\TypeHandling::getTypeForValue() instead which has the same functionality.

[BUGFIX] Only monitor YAML files in global Configuration folder

The generated IncludeCachedConfigurations.php file was also monitored for changes which actually resulted in configuration changes on each hit in development.

[TASK] Update Security Documentation

This updates the security chapter to the new security concepts.

Related: NEOS-5

[BUGFIX] Initialize Router lazily

The Router should be able to initialize configured routes lazily exept when explicitly told not to do so. This change allows the Router to get the configuration directly from the ConfigurationManager if no other routing configuration exists. If some routing configuration was set, this is used.

Fixes: FLOW-192 Fixes: FLOW-205

[BUGFIX] Support serialization of classes with static properties

When trying to serialize a proxied class that contains static properties an exception is thrown:

Runtime Notice: Accessing static property [...] as non static

The reason is that the DependencyInjection\\\\ProxyClassBuilder overrides the magic __sleep() method of the proxy class, accessing all properties as $this->\\$propertyName.

This change fixes the issue by skipping static properties.

Fixes: FLOW-282

[TASK] Make FileMonitor Strategy changes non breaking

Splits the new method setFileDeleted into a new marker interface for ChangeDetectionStrategies to allow old strategies to work without changes.

[TASK] Adjust ACL migration to cover content security

This is a follow-up to the “Restructure policy component to new Policy.yaml format” change (I84e188e89a05ec0dd1f9ee96fe312dac81806759) that adjusts the provided core migration to cover “entities” resources too. Previously a warning was shown when applying the core migration to Policy.yaml files that contained resources of type “entities”.

Related: FLOW-11 Related: FLOW-10

[BUGFIX] Repository string in the namespace break ENTITY_CLASSNAME

This change replace the call to str_replace, by preg_replace to replace only the string “Repository” at the end of the string, to avoid the bug if the Package name or any namespace segement contain the “Repository” string.

[FEATURE] Make ignoredClasses for debugger configurable in settings

Classes that should be ignored while dumping the object tree are now configurable in the settings. This makes it extendable in other packages. A default in the debugger is available, if the configurationManager is not available.

Example:

TYPO3:
  Flow:
    error:
      debugger:
        ignoredClasses:
          'Acme\\\\Test\\\\.*TestClass': TRUE

Adds a unit test for the processing of the settings Adds a functional test for merging of settings over the default

Resolves: NEOS-1140

(cherry picked from commit 33627e6d566431aaf02b473cdaf07b46c89a79fa)

[BUGFIX] Avoid Cache FileBackend tests to create actual files

Running the FileBackend unit tests, will create some test files in the current directory. This change adjusts the SimpleFileBackend unit tests in order to avoid this side-effect.

Background: This is a follow up to the “Add tests for Cache FileBackends”-change (Iee5d0de460bd7ac72018015a4cd3dad4af859868) that added unit tests that expose the side-effect.

[BUGFIX] ClassLoader should check sub namespaces

This change cleans the ClassLoader up and fixes a problem with nested autoloading namespaces. Before a nested namespace got preference and was only tried for loading a class. Now every possible path is tried for nested namespaces.

An example are the two packages:

  • guzzlehttp/guzzle
    Which declares autoloading as: "GuzzleHttp\\\\": "src/"
  • guzzlehttp/log-subscriber
    Which delcares autoloading as: "GuzzleHttp\\\\Subscriber\\\\Log\\\\": "src/"

As the second declaration was nested in the first one it got precedence for all classes inside of GuzzleHttp\\\\Subscriber\\\\ making it impossible to load classes inside that namespace from the first package. With this change both paths are tried in order of their nesting level.

Kudos go to Jens Schulze for reporting, preparing an initial fix and the test.

Fixes: FLOW-238

[BUGFIX] Test whether the current entity matches a privilege

This decouples the check, whether a query has to be modified by a EntityPrivilege SQL constraint from the generation of the actual SQL query. By this it is possible to skip not matching privileges before the SQL is generated.

One use case for this is an EntityPrivilege with a fixed entity type, where no matcher expression has to be evaluated to check the target entity type.

Additionally this adds a FalseConditionGenerator to be able to add a FALSE condition part in the SQL generation process. Furthermore this renames the AnyEntityConditionGenerator to TrueConditionGenerator respectively for consistency reasons.

[BUGFIX] Disable authorization checks in functional tests

This adjusts the FunctionalTestCase base class to disable authorization checks if the testableSecurityEnabled is not set.

Background: If security is not enable for functional tests, the Security\\\\Context is not initialized. This leads to exceptions in case the PolicyEnforcement kicks in. Because functional tests inherit the configuration of the default contexts, this happens very likely, especially when making use of node related privileges in Neos sites.

Related: NEOS-5

[BUGFIX] Simplify and stabilize security context hash calculation

This simplifies Security\\\\Context::getContextHash() and makes sure that it never returns a hash over an empty string which could be the case before.

Background: The Security Context Hash used to be generated by iterating over a list of “ContextHashComponents”. Those could be influenced using the setContextHashComponent() method. The drawback of this approach is a pretty complex mechanism that is not easy to comprehend. Besides resetting of the hash was error-prone and could lead to hash components being ignored.

This also removes the method getRolesHash() that was neither documented nor used outside of the Security\\\\Context.

Related: NEOS-5 Related: NEOS-433

[BUGFIX] Storage Object: Media type is not updated if file name changed

This changes an issue with the Resource Storage Object which might contain a media type not matching the actual file when setFilename() has been called to update the object.

[TASK] Output actual exception message for failed core migrations

Exceptions that are thrown during application of core migrations are caught in the Migrations\\Manager masking the actual error message.

This change adds the actual exception message to the re-thrown RuntimeException.

Related: FLOW-179

[BUGFIX] Fix unit test for CacheManager

This is a follow-up to the “Improve security performance” change (If6ccf8d7bbbe25cfcd6596a88a6192bf051bc9f5) fixing a unit test.

Background: The backport of this change from master to the 3.0 branch did not include all the required changes in the unit test case.

[BUGFIX] Tweaks and fixes to security:showeffectivepolicy command

This change removes some left-over references to the PrivilegeVoteResult class that no longer exists.

Besides this renames the “role-identifiers” parameter to “roles” and expands those roles in order to cover parent roles, too. Lastly this fixes support for custom privilege types allowing the “privilege-type” parameter to refer to any class implementing PrivilegeInterface.

[BUGFIX] Fix re-submission of file uploads

When using the <f:form.upload /> ViewHelper to upload a file previously uploaded resources are re-displayed via hidden fields so that they don’t have to be transmitted again. With the new Resource Management (FLOW-108) these hidden fields have been renamed from submittedFile to originallySubmittedResource (see Ic72a4ba780ebdef8b0940f3b9d20fa2c8840f834). But the ResourceTypeConverter was not adjusted accordingly.

Fixes: FLOW-214

[BUGFIX] Don’t evaluate SecurityContextHash if authorization checks are disabled

Previously Security\\Context::getContextHash() evaluated all contextHashComponents even if areAuthorizationChecksDisabled() returned TRUE. This could lead to data that was fetched within the withoutAuthorizationChecks closure to be stored in the wrong caches.

Related: NEOS-5 Related: NEOS-433

[TASK] Reset ResourceManager for functional tests

Due to the way HTTP requests are triggered in functional tests the ResourceManager was initialized very early in the process. As a result the FileSystemTarget::baseUri was initialized with an empty string.

With this change the ResourceManager is re-initialized for every test.

[TASK] Improve security performance

This solves two problems. First it eliminates the need to use eval for AOP runtime expressions by caching the found expressions in a PhpBackend during compile time and requiring the code as actual closures during runtime. This also reduces the cache size as the closure code is no longer included as strings.

Additionally the methodPermissions data is changed to be static as it will be the same across all instances of MethodPrivilege anyway. This drastically reduces memory usage and also runtime due to less cache requests which are especially costly for a file based cache.

[BUGFIX] Only monitor files with relevant file extensions

This change adjusts the ClassFiles, ConfigurationFiles and TranslationFiles file monitors to only track changes in files with the corresponding extension (php for classes, yaml for configuration and xlf for translations).

Background: With the fixed PSR-4 support (introduced with I9c613df54a8b650c53b4ab8e03071432e13d3c4e) we no longer determine PHP classes from the corresponding file path and -name, but by analyzing the contents of the file. Because the Flow_ClassFiles file monitor watched for changes in all files within the classes root of all packages, this could lead to binary files being analyzed. In some PHP versions this seem to trigger a warning.

Fixes: FLOW-271 Related: FLOW-238

[!!!][TASK] Use a persistent cache for storing encryption key

This change takes advantage of the new persistent caches feature and stores the encryption key used by the Hash Service in such a persistent cache. If an EncryptionKey file exists in Data/Persistent from earlier Flow versions, that file will be used instead.

This change may require your attention if you tailored your automated deployment to the encryption key file being located in Data/Persistent.

[!!!][FEATURE] Persistent Caches

This change introduces the concept of persistent caches. Caches which are configured to be “persistent” will be skipped by the Cache Manager when flushing all caches or flushing caches by tag. Therefore it is now possible to use the caching framework as a low level key-value-store for storing keys, preferences, tokens or other artifacts which usually would have been stored in the Data/Persistent directory.

If persistent caches are used consequently, it is possible to configure a Flow application to not store any persistent data in the file system. This allows for clustering and more flexible management in cloud environments.

A cache is marked as “persistent” by setting the respective flag in the configuration in Caches.yaml:

Acme_MyCache:
  frontend: TYPO3\\Flow\\Cache\\Frontend\\StringFrontend
  backend: TYPO3\\Flow\\Cache\\Backend\\SimpleFileBackend
  persistent: true

Persistent caches won’t be flushed by the flow:cache:flush commands, and for file based caches, the data is not stored in Data/Temporary/Cache by default, but in Data/Persistent/Cache.

Note: This change might break unit tests which mock cache frontend / backend combinations because a new lifecycle method (initializeObject) has been introduced to the AbstractFrontend. In order to fix these tests, simply make sure to call initializeObject() after the frontend has been instantiated.

Resolves: FLOW-261

[TASK] Add tests for Cache FileBackends

Besides adding unit tests covering the basic functionality of the SimpleFileBackend this change introduces tests for the recently modified way of including PHP files via SimpleFileBackend::requireOnce().

Background: Previously files where included within a try/catch block swallowing errors in included PHP files. With Ied1dd72d4d9035a6f7673ceb096cc908c767d80c the try/catch block was removed for performance reasons, with the side-effect that errors and exceptions are no longer silently skipped.

Depends: Ied1dd72d4d9035a6f7673ceb096cc908c767d80c

[BUGFIX] Properly expand parent roles in Policy.yaml core migration

The core migration TYPO3.Flow-20141113121400 that rewrites existing Policy.yaml files to match the reworked Security Framework also adjusted the role identifiers that were allowed to only contain the role name without package key prefix if used in the current package.

With this fix the role identifier expansion now also works for referenced parent roles.

Fixes: FLOW-221 Related: FLOW-11

[!!!][BUGFIX] Make bidirectional OneToOne relations work correctly

This change fixes bidirectional OneToOne relations, which previously resulted in foreign keys being created on both sides. Therefore the related entities could no longer be deleted.

The problem was that the joinColum mapping was created in every case, which doctrine interprets as a secondary hint for the owning side. The joinColumn should only be created for the owning side, i.e. only for unidirectional OneToOne and for the inversedBy side of a bidirectional OneToOne.

This change is not breaking, but needs manual attention to fix the schema of own projects as follows:

After upgrading to this change, you should run following commands from command line to update your projects:

flow doctrine:migrationgenerate
-> check the created migration file, it should contain a number of
“DROP FOREIGN KEY”, “DROP INDEX” and “DROP {column}” statements for each of your bidirectional OneToOne relations.
flow doctrine:migrate
-> applies the changes

Fixes: FLOW-87

[TASK] Move a unit test fixture class out of test class

A trivial change, but it is cleaner and allow to use brianium/paratest for running the unit tests.

[TASK] Better configuration error handling in EntityManagerFactory

The Doctrine\EntityManagerFactory in Flow gets slightly better error handling covering configuration of Doctrine and backend options.

[TASK] Store nicely formatted JSON with JsonArrayType

[!!!][TASK] Remove compileConfigurationFiles setting

Since the configuration cache is now cleared when changes are detected, the possibility to actually enable/disable that caching is no longer needed.

[TASK] Only flush AOP proxy classes if needed by config change

This change does no longer remove the AOP proxy classes upon every change to configuration, since e.g. Views.yaml or changes to NodeTypes in Neos do not need a proxy class rebuild.

With this only changes to Settings, Objects and Policy configuration flush the AOP proxies.

[TASK] Flush configuration cache through file monitor

If the file monitoring detects changes to configuration files, flush the configuration cache.

With this in place, the change also removes the settings for Development and Testing context that switched caching off. This improves performance in those contexts.

[TASK] Cleanup FileMonitor

This does a cleanup of the FileMonitor at the same time improving performance. Reading the current state of directories is now done with a Generator method to reduce memory footprint. Additionally the amount of looping was reduced.

Finally you can now exclude whole subdirectories by adding an empty file named .flowFileMonitorIgnore to it.

[TASK] Optimize garbageCollection in FileBackend

We can retrieve the cache identifier for a given cache file much easier with the directoryIterator than is done currently.

[TASK] Optimize requireOnce in SimpleFileBackend

We go back using a is_file check before trying to require a cache file. This is the safest way to detect existence and is_file is still faster than file_exists.

This also prevents errors that happen because a previous exception in file requires was caught.

[BUGFIX] The EntityPrivilegeExpressionEvaluator is singleton

The EntityPrivilegeExpressionEvaluator is already annotated as scope singleton but was instanciated in the EntityPrivilege. That is quite costly with a lot of queries. Now the ObjectManager is asked for an instance of the evaluator.

[BUGFIX] Fix PSR-4 support for Flow packages

Basic support for PSR-4 structured packages has been added with Flow 2.2 (see I9b2dae7761ef48389d9915c1269df2fdf771af8c).

However, automatic re-reflection didn’t work for PSR-4 Flow packages. Besides, custom Package.php files were expected to be located at Acme.PackageKey/Classes/Acme/PackageKey/Package.php which is not in sync with PSR-4.

This change fixes the two issues by extracting the actual class name from the actual PHP files instead of trying to defer it from the file path. Besides it replaces the static lookup path for custom Package.php files taking the autoload property of the composer manifest into account.

Note: While this fixes basic support for PSR-4 Flow still doesn’t support all composer features. Primarily there is a 1:1 mapping from package to autoloading type and classes path while composer manifests support multiple mappings by namespace.

Related: FLOW-238

[TASK] Use doctrine caching in functional tests

Enable doctrine caches for functional tests to avoid slowdowns due to recalculation of Query filters. Two tests clear the caches to avoid side effects.

[BUGFIX] ValidatorResolver shouldn’t ask class loader for simple types

The ValidatorResolver generates a Validator conjunction for a given class if that class exists. Problem is that there is no check for simple types before this, so a simple type will trigger the class loader (every time). To avoid that we now check if the given type is a simple type.

Related: Icefc6eadbe7cae2e12cc2204ec6de3f25370aefa

[BUGFIX] Setting properties from Objects.yaml should invoke setter

If you configure simple values via Objects.yaml they should be set via setter if one exists, as the setter could manipulate the incoming value. Without this change the injected values are directly set to the property circumventing the setter.

Fixes: FLOW-30

[BUGFIX] Fix “Call to undefined method ...::getPrototype()”

When injecting dependencies that take prototype-scoped objects as constructor parameter, the ProxyClassBuilder class tries to instantiate these objects using the getPrototype method. This method is apparently supposed to be implemented by the proxy class, but never is.

This results in a fatal error when the owning object is instantiated:

Fatal error: Call to undefined method My\Class::getPrototype() in ...

This commit fixes this behaviour by replacing the getPrototype call with a simple constructor call and adds a (kind-of) simple functional test for this behaviour.

Fixes: FLOW-175

[BUGFIX] Add format to encoded DateTime objects in JsonArrayType

To be able to convert the stored DateTime back into the object we add the format as well. The stored array can directly be used with a DateTimeConverter.

Related: I7eeb7d55e8ee593cae5bccf98211e4a2ff83b1d8

[FEATURE] JsonArrayType for doctrine persistence

This adds an extended JsonArrayType that can en-/decode Flow entities. It is not used anywhere for now.

Related: NEOS-427

[BUGFIX] Persisting of objects as serialized data not supported

The ReflectionService does not add properties to metadata information of classes if they are not tagged with @Flow\\Entity or @Flow\\ValueObject. This makes it f.e. impossible to store an object as serialized object. Previously any object (except some whitelisted) needed at least a @ORM\\OneToOne annotation to be persisted in a property.

To be able to store f.e. UserSettings, BasketItems, Sessions or DTOs the doctrine Annotation @ORM\\Column(type="object") is now supported.

The whitelist was removed from ReflectionService in order to allow any class to define custom column types.

The ClassSchema does no longer contain information about properties declared @Flow\\Transient, the same is true for properties annotated with @Flow\\Inject so that injected properties are never persisted.

If an object has a property which holds an object and that property is annotated with @ORM\\Column(type="object") that will be stored as expected (i.e. as a serialized PHP object). If no such annotation is given and no other relation type annotation is given, an exception is thrown as before.

Working Example:

/**
* @var \\Acme\\Foo\\Bar;
* @ORM\\Column(type="object")
*/
protected $bar;

In the example above the doctrine behavior for object types matches and invokes \\Doctrine\\DBAL\\Types\\ObjectType

Also added some tests for other extended types supported by Doctrine:

  • \Doctrine\DBAL\Types\Type::SIMPLE_ARRAY
  • \Doctrine\DBAL\Types\Type::JSON_ARRAY
  • \Doctrine\DBAL\Types\Type::DATETIME
  • \Doctrine\DBAL\Types\Type::DATETIMETZ
  • \Doctrine\DBAL\Types\Type::DATE
  • \Doctrine\DBAL\Types\Type::TIME

Please note:

Fixes: FLOW-254

[BUGFIX] Don’t store SecurityContextHash in session

Adds Flow\\Transient annotations to Security\\Context::contextHash and Security\\Context::contextHashComponents so that they won’t be stored in the session and refreshed for every request.

Related: NEOS-433

[TASK] Refactor initialization of security operations in behat tests

This change leave the decision, if security should be initialized or not to the respective behat context. As tests executing code, which uses security features all need an initialized security component, we cannot initialize only when steps of the security trait are called in the tests explicitly.

[BUGFIX] Don’t skip core migrations quietly

This change makes sure that ./flow core:migrate always informs the user about skipped packages even if the verbose flag is not set.

Background: This is a follow up to the “Improve handling of core migrations” change (If66a2dff21b239963728963f15437599a8442f72) that introduced this new behavior.

Related: FLOW-179

[BUGFIX] Whitelist added objects to resource repository

When an image variant or thumbnail could not be generated during the import of the image. The generation on the first GET request failed also due to the missing whitelisting.

(cherry picked from commit 5c7e3a803f627389c3718fd4cd8008517f791344)

[BUGFIX] Re-activate authorization checks only when not disabled already

This change fixes the re-enabling of authorization checks, when calling Security\\Context::withoutAuthorizationChecks() in nested calls.

Fixes: FLOW-249

[BUGFIX] Remove Persistent Resource Rewrite Rules and Update Documentation

For Flow 3.0 and the new Resource Management, these lines in the config are not needed anymore; in fact, they even harm because they prevent file uploads from working correctly.

Resolves: FLOW-246 Related: NEOS-1111

[TASK] Add a place for release notes to documentation

This adds a home for release notes to the documentation.

[TASK] Proper(er) solution for connection issue hotfix

Right before the release of 3.0.0 beta 1 a hotfix was added to enable proxy compilation even if no database connection is possible. Said problem is caused by the update to Doctrine ORM 2.4 which pulls in a new DBAL version as well. That offers version-aware platform handling and in turn tries to connect and read the server software version used whenever the database platform instance is fetched.

This change drops the unconditional try/catch hotfix and replaces it by more (but not less awkward) code. If a connection is not possible, the generic platform tied to a driver will be used, which is sufficient for the tasks that make sense without a connection.

[BUGFIX] Fix error in PostgreSQL down migration

In the down migration of Version20141113145146 the dtype column was added again, even though it was already present (from the previous down in Version20141118174722).

The removal of the dtype column in the up migration is removed as well, since the whole table is dropped in the next up anyway.

[TASK] Add “cleanup” schema migrations

This adds migrations for MySQL and PostgreSQL that tweak the DB schema a bit, cleaning up some leftover discrepancies.

[TASK] Tweak schema migrations to Flow/Party decoupling

This adjusts schema migrations to that it is actually possible to:

  • install Flow without Party
  • add Party to a “clean” Flow setup

Related: FLOW-5

[TASK] Add @covers to list of ignored annotations

With recent PHPUnit versions phpdocumentor/reflection-docblock is pulled in. This contains a @covers annotation that triggers an exception during reflection. To avoid this, it is added to Flows default list of ignored annotations.

[TASK] Adjust unit tests to “Flush compiled configuration”

The reworking of the change Ic3af97d652e87271ed7c2dfbc1091ea4cb7e1973 forgot to adjust the unit tests. This reverts the needed changes.

Related: FLOW-229

[BUGFIX] Flush compiled configuration on cache:flush

The ConfigurationManager writes compiled configuration to disk (if this is enabled, by default this is the case in Production context). But this compiled configuration is not removed when flushing caches, so that changes never become active.

This change adjusts the CacheManager to be aware of this compiled configuration and remove it during flushCaches().

Fixes: FLOW-229

[TASK] Remove unused injection of Doctrine EntityManager

The Doctrine EntityManager is injected in the ResourceManager, but never used.

Additionally removes the unused protected property importedResources. The getter (which is tagged @api) is still there but already gets the information from the ResourceRepository.

[BUGFIX] Fix resolving of imported interface namespaces

Dependency injection and property mapping does not work for interfaces of the current namespace if the code does not refer to the fully qualified name.

Fixes: FLOW-34

[TASK] Bump version in Documentation/Settings.yml

[BUGFIX] Resource should not unpublish if file is in use

This change fixes an issue with the Resource Manager’s deleteResource method which unpublished the respective resource data even if it was still in use by other ressources.

Additionaly optimise resource deletion so that the resource manager will only unpublish / try to unpublish resources once. Previously deleteResource was triggered through a direct method call (for example by an ImageVariant) and then additionally through the Doctrine lifecycle event.

[TASK] Tiny fix to TranslationHelper docblock

Simply fixes a copy-and-paste error.

[FEATURE] Make custom Doctrine mapping types configurable

This removes the hardcoded registration of the ObjectArray mapping type and instead introduces mapping type configuration from settings:

TYPO3:
  Flow:
    persistence:
      doctrine:
        # DBAL custom mapping types can be registered here
        dbal:
          mappingTypes:
            'mytype':
              dbType: 'db_mytype'
              className: 'Acme\\Demo\\Doctrine\\DataTypes\\MyType'

See the Doctrine documentation for more details: http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/cookbook/custom-mapping-types.html

[TASK] Add changelog for TYPO3 Flow 2.3.2

See https://ci.neos.typo3.org/job/typo3-flow-release/37/

[BUGFIX] Fix hostname option name for Redis backend

Redis backend option for specifying a hostname is not host but hostname.

[BUGFIX] Don’t append subrequest arguments to toplevel request

For sub-requests the Uribuilder would append the sub-request arguments again without namespace prefix if addQueryString was set. This can lead to invalid URLs and possible side-effects. An example of this would be that having this URL:

some/action?namespace[argument]=foo

And building a URL in the subrequest with addQueryString enabled and arguments set to array('argument' => 'bar'). Previously the resulting URL was:

some/action?namespace[argument]=bar&argument=foo

With this change it is:

some/action?namespace[argument]=bar

Fixes: NEOS-1027

[BUGFIX] Pointed the issue tracker URL to the new JIRA instance

[BUGFIX] Do not register lifecycles for unproxied entities

When entities are annotated @Flow\Proxy(false), lifecycle methods for fix* are still registered in the AnnotationDriver, leading to an exception being thrown by doctrine that the lifecycle callback method does not exist.

Since unproxied classes are not supposed to work with AOP/DI and those fix* methods only deal with that, the registration of those fixes is unneeded in that case.

This changeset fixes that, by only registering lifecycle callbacks on entities that are proxied.

Fixes: FLOW-258

[FEATURE] Request patterns for host name and IP addresses

This change adds RequestPatterns for host name and IP addresses. Host names are configurable with a simple wildcard (‘*’) pattern, IP addresses use CIDR style pattern with support for both IPv4 and IPv6 following RFC4632 and RFC4291.

Examples:

Only grant access to clients with IPs from 192.168.178.0 to 192.168.178.255:

TYPO3:
  Flow:
    security:
      firewall:
        rejectAll: TRUE
        filters:
          'ip':
            patternType:  'Ip'
            patternValue: '192.168.178.0/24'
            interceptor:  'AccessGrant'

Protect an area of your application file based authentication and IP check (only trigger authentication and entry points for clients with an IP of 127.0.0.1):

TYPO3:
  Flow:
    security:
      authentication:
        providers:
          'LocalNetworkProvider':
            provider: 'FileBasedSimpleKeyProvider'
            providerOptions:
              keyName: 'AdminKey'
              authenticateRoles: ['Acme.SomePackage:Administrator']
            requestPatterns:
              controllerObjectName: 'Acme\\SomePackage\\Controller\\Administration\\.*'
              ip: '::7F00:1'

[BUGFIX] First Functional test run should not be marked risky

Changing globals inside of the test code (which includes the setUp method) will set the test to risky and immediately skip the execution of the test. Globals are now changed in setupBeforeClass to avoid this problem.

[BUGFIX] Check if migrations exist before trying to register them

Currently the check for the existence of migration files is delegated to the doctrine/migration third party library.

The behavior has changed and an exception is thrown if the folder doesn’t exist. To not let the third party library handle that and to prevent this, the check is done upstream in Flow.

Fixes: FLOW-301

[TASK] Require doctrine/migrations 1.0.*

With doctrine/migrations not having had a stable branch for years, Flow always required dev-master. Now that 1.0 is being worked on, require that instead.

Related: FLOW-300

[BUGFIX] Save unicode characters unescaped in JsonArrayType

As there might be the need of a string search in the JSON data all unicode characters should be stored unescaped to allow them to be properly searched and compared.

[BUGFIX] Reset class loader after recompiling proxies

The class loader holds a list of available proxy classes to prevent unnecessary calls to the proxy class cache. This list is loaded early in the bootstrap and needs to be refreshed when a sub request changed the proxies.

Fixes: FLOW-309

[BUGFIX] Support composer replace directive

Packages that are required but were replaced by using replace (see https://getcomposer.org/doc/04-schema.md#replace) in composer.json would to load with Flow as the replaced package would have no name in the package map. Ignoring the dependency is no problem as we do not use it anywhere where replaced packages could be important.

This is a bugfix that should be replaced with a more robust solution in a future Flow version. For example by not trying to match composer requirements to Flow packages.

Fixes: FLOW-287

[BUGFIX] Filenames need to be rawurlencoded

The new resource management does not urlencode filenames of resources in URLs. This leads to invalid URLs for file paths/names containing special characters or spaces.

This patch fixes the issue by applying rawurlencode() to all path segments in FileSystemTarget::getPublicStaticResourceUri() and FileSystemTarget::getPublicPersistentResourceUri()

Fixes: FLOW-321

[BUGFIX] Use rawurl(de|en)code in path segments

According to RFC 3986 the space character has to be percent encoded (%20) which is correctly done by rawurlencode. Using urlencode will lead to spaces being encoded as + signs which is technically incorrect for path segments and should only happen for query string values.

[TASK] Decrease session GC probability to 1% (was 30%)

Garbage collection of expired sessions should occur less often since it’s trigger for every third request right now on average. A 1% value matches the PHP default value and should be sufficient for most setups.

Resolves: FLOW-146

[BUGFIX] 1st level cache for QueryResult::count()

Counting the results of the QueryResultInterface proxy that is returned by Repository finder methods always executed a COUNT query, even if the results have been fetched or counted already.

With this patch the number of results is cached as soon as the result is initialized and/or counted once.

Fixes: FLOW-326

[TASK] Update “Quickstart” tutorial

Adjusts the Quickstart tutorial to be in sync with recent Flow changes.

Resolves: FLOW-298

[TASK] Support dynamic return types for PhpStorm IDE

This adds configuration for the PhpStorm IDE so it can infer the return type of the get() method in ObjectManagerInterface.

[BUGFIX] Accept incoming HTTPS headers

This removes a check from Http\\Headers that prevented any header starting with “HTTP”.

Background: A recent version of Google Chrome seems to send a HTTPS header for requests via SSL (see https://code.google.com/p/chromium/issues/detail?id=501095). This header translates to $_SERVER['HTTP_HTTPS'] which triggered the exception.

Fixes: FLOW-324

[TASK] Tweak doc comments in two classes.

[BUGFIX] Failing unit test in XliffModelTest

Regression of I3b5f7e06eca6a2e7e7aa31148c1d031295964c92 which didn’t adjust the test accordingly.

[TASK] Lower log level for I18n sub package

This change lower the log level of the I18n package from LOG_WARNING to LOG_DEBUG.

[BUGFIX] Clear PHP Opcode Caches in Caching, PackageManager and ConfigurationManager

This solves a lot of bugs when people use a wrongly configured opcode cache (e.g. part of Bitnami MAMP stack).

Check the ticket for further background and how I could reproduce the error on Bitnami MAMP stack.

Thanks to the TYPO3 CMS project for inspiration on the solution!

Resolves: FLOW-320

(cherry picked from commit ad3b949d65dc2f1d585a28119452692a153f2f4f)

[BUGFIX] Make resource:// URIs work with 40 character Package Keys

The Flow ResourceStreamWrapper transparently handles resource URLs in the formats resource://<SHA1> (referring to the SHA1 hash of a Flow Resource) and resource://Package.Key/some/path (referring to a local file path relative to the specified package’s Resources directory).

Currently the detection of a SHA1 hash is unreliably only checking for the string length being 40 characters. If a Package Key happens to be exactly 40 characters long, static resources can’t be loaded. This change fixes the issue by replacing the strlen check with a simple regular expression lookup that only accepts valid SHA1 hashes.

Fixes: FLOW-250

[BUGFIX] Fix warning about undefined index in Autowiring

The Autowiring annotation did not check the existence the existence of the value key, leading to a warning, followed by an exception, when the annotation was used like:

Autowiring(enable=false)

instead of:

Autowiring(false)

and the ReflectionService was asked for reflection data on such a class.

The same problem could affect Proxy annotation, so it is fixed as well.

[BUGFIX] Prevent creation of too many directories for resources

Currently resource files are stored by splitting the SHA1 hash into chunks of 5 characters in order to prevent the creation of too many files per directory (for performance reasons). This has been implemented with the new “Multi-Storage / Multi-Target Resource Management” (see Ia2b47b4070a2dfabf4833bf1f0f3967ba3b032a7).

The drawback of the current implementation is that it leads to the creation of a directory per file which strains the file system, too.

With this change a persistent resource will no longer be stored in:

Data/Persistent/c828d/0f88c/e197b/e1aff/7cc2e/5e86b/12442/41ac6/c828d0f88ce197be1aff7cc2e5e86b1244241ac6

But:

Data/Persistent/c/8/2/8/c828d0f88ce197be1aff7cc2e5e86b1244241ac6

leading to a tremendously reduced amount of folders to be created.

This also adjusts the path of published resources from:

/_Resources/Persistent/c828d/0f88c/e197b/e1aff/7cc2e/5e86b/12442/41ac6/The-File-Name.ext

to:

/_Resources/Persistent/c/8/2/8/c828d0f88ce197be1aff7cc2e5e86b1244241ac6/The-File-Name.ext

Note: This is only true, if the subdivideHashPathSegment is set for the corresponding PublishingTarget. Otherwise the path is not changed.

Related: FLOW-294

[TASK] Show expected targetType on “Object not found” errors

If the PersistentObjectConverter cannot find an object by given identity the error message shows the identity but not the expected oject type. Adding it helps a lot in debugging such errors.

[BUGFIX] Allow explicit privileges for Everybody role

The TYPO3.Flow:Everybody role that is always active by design gets an ABSTAIN permission for all defined PrivilegeTargets so that permission to restricted privileges is not granted by default.

Due to a bug this ABSTAIN permission was also set when a privilege was explicitly configured for the Everybody role as soon as the same PrivilegeTarget was used for another role, effectively overriding the permission.

This change allows for explicitly allowing or denying privileges to the Everybody role. It also adds some unit tests for the PolicyService.

Fixes: FLOW-308

[TASK] Add missing changelogs

[TASK] Fix rST rendering warnings

[TASK] Show version and release in front matter

[TASK] Simplify appendix TOC generation

[TASK] Rename Index.rst to index.rst

This is our workaround for an issue with rtfd.org so the documentation can be browsed when it has been rendered there.

(See https://github.com/rtfd/readthedocs.org/issues/1279 for details.)

[TASK] Switch to plain Sphinx build configuration

Switches to a more standard setup for rendering rST using Sphinx. A simple make html renders the documentation, if sphinx is installed.

The conf.py contains a tweak to allow using the RTD theme without confusing the renderer over at RDT itself.

[BUGFIX] Relax YAML schema to work with PECL parser

This adjusts the Objects.yaml schema so that:

./flow configuration:validate

does not complain about:

Objects.TYPO3.Flow.DateTime.autowiring -> expected: type=string found: type=boolean

Background: The PECL YAML parser interprets the strings “on” and “off” as booleans so we need to adjust the schema accordingly.

[BUGFIX] ReflectionService should use normalized simple types

The method parameter and property type reflection should contain type names for simple types that are already normalized so you can rely on that in comparisons later on. The method expandType now normalizes the type if it detects a simple type.

[BUGFIX] Adjust YAML schema to recent changes

This is a follow-up to the “Make ignoredClasses for debugger configurable in settings” feature (If4a692168d317cbcb7527fb28ebb113705f7c500) adjusting the YAML schema.

Related: NEOS-1140

[BUGFIX] Adjust YAML schema to recent changes

This is a follow-up to the “Centralized Neos user domain service” feature (I8a4c7010e25842583538e32dc8ef9cd851f306b4) adjusting the YAML schema.

Related: NEOS-962

[BUGFIX] Adjust YAML schema to recent changes

This is a follow-up to the “Make custom Doctrine mapping types configurable” feature (Iab0f14117b30a4924a7947af44b9516c241912da) adjusting the YAML schema.

This also tweaks the existing schema rules according to our CGL.

[BUGFIX] Adjust YAML schema to “Persistent Caches”

This is a follow-up to the “Persistent Caches” feature (I0e18122d74807a75c6e544c5074c8e924e21c649) adjusting the YAML schema.

Related: FLOW-261

[BUGFIX] EntityPrivilege expressions should be trimmed

The EelEvaluator trims expressions before parsing as spaces at the beginning or end might break the parser for no good reason. Same is done in the EntityPrivilegeExpressionParser now.

[BUGFIX] Controller changes trigger AOP class rebuild

This is a pragmatic fix to avoid clearing all caches when you add an action to a protected controller. Controllers will now be watched by the FileMonitor to trigger rebuilding the policy information.

[TASK] Improve PackageManager speed

The main improvement is the addition of a first level cache for composer manifest data that was read once to avoid reading the files multiple times.

Additionally the package path is reused in the PackageFactory to reduce the amount of concatinated path segments.

[TASK] Use StringFrontend for HashService cache

The use of StringFrontend was intended, but not configured in Caches.yaml, so a VariableFrontend was used instead for the persistent cache.

[FEATURE] Color support for core migrations

This change allows core migrations to output colored texts. This is especially useful for errors and warnings that are likely to be overlooked due to the verbosity of core migration output.

Note:

This feature requires the posix PHP extension to be active (ANSICON/ConEmuANSI support on Windows).

[BUGFIX] Allow core migrations to be applied to non-git-repositories

Allows core migrations to be applied to packages that are not under git version control.

With this change the log for core migrations is no longer tacked through the git log history but with a new section applied-flow-migrations in the extra property of the composer manifest. This allows for keeping track of the core migration log independently from a git repository.

The git log migration history is still respected and will be imported into the composer.json file of a package as soon as core migrations are executed.

This change also introduces a new flag --force that can be specified in order to apply migrations even if the affected package is not part of a git repository or contains local changes.

Background:

This is not marked FEATURE because it fixes a severe bug that prevented migrations to be applied when having multiple packages in the same git repository.

[BUGFIX] Value objects get cascade deleted by default

Value objects are by nature not bound to single parent entities and exist by their own. Currently, Flow automatically annotates relations to value objects as cascade=’all’ and orphanRemoval=true. Doctrine itself states that orphanRemoval should only be set when the related object is privately owned and not reused by other entities, which is not generally true for value objects. See http://doctrine-orm.readthedocs.org/en/latest/reference/working-with-associations.html#orphan-removal

This leads to exceptions when entities with a relation to an value object are deleted.

This change addresses that by setting cascading to ‘persist’ only for value objects and not setting orphanRemoval on relations to value objects by default.

[BUGFIX] Tabs should be allowed in front of classes

Currently the proxy class Compiler uses a regular expression that only allows spaces before the class keyword. So a class declaration:

abstract<TAB>class Example ...

will not be correctly detected and compiled. Changing the expression to allow any kind of whitespace character fixes these edgecases.

[BUGFIX] Fix permission of temporary files when importing resources

When importing a site on Microsoft Windows 7/8 an exception can be thrown due to insufficient file permissions. This change makes sure that the file permissions are fixed before a temporary resource file is moved to its final location.

(cherry picked from commit 3cdac302dd1d2e04346493cb9926a2ade82ce01a)

[BUGFIX] Don’t skip schema validation for empty configurations

When performing a configuration validation via:

./flow configuration:validate

The validation was interrupted with an exception if one of the parsed configuration types is empty (for example because no Views.yaml file existed). This patch fixes the issue by replacing the exception with a notice that can be outputted with the verbose flag.

(cherry picked from commit 3d937c392ef6dd15382ad442b63cf411d999d132)

[BUGFIX] Do not cast NULL to empty string in PropertyConditionGenerator

Due to the escaping of NULL values, the resulting SQL query did a check against an empty string, not NULL values.

The PropertyConditionGenerator now holds the raw parameter values to check against NULL values.

[TASK] Add example configuration to disable the routing cache

While developing route part handlers it is really useful to disable the routing cache temporarily. This change adds an example configuration in composer essentials installed using the installer. This can then easily be used to create a local Caches.yaml

[BUGFIX] Relax YAML schema to allow custom requestPatterns

When using the ./flow configuration:validate command to verify that the settings conform to the configuration schema, Flow complains with:

Settings.TYPO3.Flow.security.authentication.providers.Typo3BackendProvider.requestPatterns.<pattern>
-> This property is not allowed here, check the spelling if you think it belongs here.

when using custom requestPatterns.

This change relaxes the YAML schema to allow arbitrary requestPatterns.

(cherry picked from commit d40e4b349c893580a10fda379358ad8bc8aaf0d3)

[BUGFIX] Allow Aggregate Roots to extend non entity classes

When an entity with a repository (= Aggregate Root) extends a Plain Old PHP Class (no entity) the ReflectionService produced a Undefined index notice when building the Class Schema.

This change fixes this by skipping parent classes that are not persistable.

[BUGFIX] Prevent depletion of inodes during migration

Instead of moving all files to the legacy location, creating many folders, the migration now moves resources directly to the final destination.

The second migration is a fallback for Flow installations that are not fully migrated yet.

[BUGFIX] When moving resource files, check if expected file exists

The moving of resource files in migrations Version20150611154419 and Version20150611154421 did always happen, even if the expected target did already exist. This lead to bogus warnings when a file was used by more than one Resource instance.

On top of that, empty directories were never properly removed.

[BUGFIX] Better moving of files in migration

The migration that moves resource files around (Version20141118174722 for PostgreSQL) did not consider the case of multiple resources using the same file. This change adjusts the code so it behaves like the corresponding migration for MySQL, which has better handling for this.

[BUGFIX] Correctly close all streams opened from a Resource

The resource opens a stream (e.g. file handle for the file storage) when calling getStream() using the underyling storage. It’s the callers responsibility to close the stream after consuming the content.

This change adds some missing fclose() calls and adds an error check to the ArrayConverter that results in better error messages when dealing with broken resources.

[BUGFIX] Re-enable logging of Router::route() calls

With the implementation of HTTP Components we accidentally disabled logging for calls to Router::route().

This patch adds the log() calls again.

Background:

The regression has been introduced with “[FEATURE] HTTP components for handling requests” (Iac1bd27cd1f2869e597b696c896633f14703ec40).

[TASK] Do not proxy MethodPrivilege for performance

As the method privilege is used rather often in a request even simple method calls add up. The (unecessary) proxy for the MethodPrivilege adds some overhead in form of array_key_exists calls in the __constructor inherited from the AbstractPrivilege. Disabling the proxy removes this overhead.

[TASK] Make PHP the default language for code blocks

The documentation now uses PHP as default language for code blocks, so that even when using just two colons to start a code-block it is highlighted.

[BUGFIX] Behat Subprocess command should be properly escaped

The behat subprocess command should be properly escaped to work with paths that contain special characters.

[BUGFIX] Throw 404 exception when updating a non-existing resource

Specifying a non-existing identity when converting a persistent object using the PersistentObjectConverter the PropertyMapper tries to apply conversions on the TargetNotFoundError instead of returning the error directly.

Background: The reason for this regression is the new TargetNotFoundError that has been introduced with I5231ccddaef1a6b49e62bb29bdd3ff99dc2bb994 in order to be able to handle that case more gracefully. Previously an exception was thrown directly.

[TASK] Add support for sphinx-autobuild in docs

This adds support for sphinx-autobuild, which watches the Documentation directory and automatically re-renders docs when changed. This allows for a livepreview while editing the docs. It also serves the pages via the python-livereload server so they will be automatically refreshed when the docs are re-rendered.

To use it, install sphinx-autobuild:

pip install sphinx-autobuild

To use livepreview just run make livehtml instead of make html. Then, visit http://127.0.0.1:8000 to see the livereload version of the docs.

make livehtml renders the docs exactly the same as make html does. A livereload.js script does get injected, but not by the rendering process. The script gets injected on the fly by python-livereload when serving the pages.

[TASK] Make PHP highlighting work for snippets

This adds the “official” hack to turn on startinline for all php codeblocks whether they’re a snippet or a full file.

Sphinx and docutils don’t support setting the startinline option for pygments. That means that snippets of PHP code that don’t have “<?php ?>” don’t get highlighted. The hack was documented here: http://mbless.de/blog/2015/03/02/php-syntax-highlighting-in-sphinx.html https://github.com/fabpot/sphinx-php

[TASK] Use a separate logger for I18n and missing translations

This change introduces a separate logger for I18n related messages, especially when translation labels are missing.

[TASK] Merge Eel I18n helper into TranslationHelper in Flow

The I18n helper cannot reside in EEL as it creates a hard dependency on Flow that we want to avoid. Instead it should be merged into the TranslationHelper in Flow that already existed.

[!!!][TASK] Increase PackageStates version

Flow 3.0 depends on some new 3rd party package versions. Because the name of some of those packages have been changed the casing, this leads to exceptions like:

Package "doctrine.instantiator" is already registered as "Doctrine.Instantiator"

By increasing the version, the PackageStates.php file will be re- generated automatically, removing obsolete package keys.

This is a breaking change because it will re-activate any previously disabled packages! If you want to avoid that, temporarily move affected packages into a separate directory or disable them after upgrading.

[BUGFIX] Fix site locking

Site locking is prone to race conditions and should initialize much earlier in the request and use a reliable locking method.

This change moves the Locking to be beginning of the Bootstrap Sequence and changes the locking method to use flock which is less prone to race conditions.

[BUGFIX] PersistentObjectConverter should correctly detect class

The PersistentObjectConverter needs to know the class of an object after conversion, as the object can be a doctrine proxy the use of get_class is not correct and replaced by the TypeHandling utility which resolves the correct name even with a doctrine proxy.

[TASK] Remove ChangeLogs for other branches from documentation

Having the ChangeLogs of all previous branches only balloons the documentation without a real gain.

[TASK] Move Doc Config from DocTools package

This moves the doc configuration from the DocTools package to the Flow package so that it can be changed per branch. To see the DocTools change, refer to I9b56df70f6b5140b41b41893a8fb6f516b8f1843

[TASK] Update “Getting Started” tutorial

Adjusts the “Getting Started” tutorial to be in sync with recent Flow changes and best practices.

[BUGFIX] Respect configured “baseUri” in resource URIs

As all resource targets are initialized during bootstrap, the initalizeObject lifecycle method is called before the http request has any configured baseUri set, therefore the absolute URI did not respect the configuration setting. By resolving the baseUri lazily we can avoid that.

[BUGFIX] Cache flush force should be possible in all contexts

Force flushing caches is a compiletime action and as such could trigger locking of the site in production context. This leads to strange locking errors which were not planned for. With the change to fix it I8152343764795c4c5a407547d6094d16655c7b32 a regression was introduced as the LockManager is only available in Production context so an exception was triggered in other contexts when force flushing caches. With this change the context is checked before trying to access the LockManager.

[BUGFIX] Unlock site after force-flushing caches

[TASK] Fix version in conf.py

TYPO3.Fluid

[TASK] Undo requirements adjustment from CI job

The release CI job adjusted requirements dutifully, but the result was not as expected, because we humans missed some needed changes.

[TASK] Mark recent core migrations applied

This commit doesn’t contain any changes, it simply marks recent migrations applied so that:

./flow flow:core:migrate --status

won’t show any open migrations for this package.

[BUGFIX] Fix unit test in AbstractFormViewHelperTest test case

This is a follow-up to I80e7e664e7a1fa41dc36bdf89e331086c6815f78 that fixes a unit test.

Related: FLOW-213

[BUGFIX] Fix retrieval of property mapping results if formObjectName is not specified

This is a follow-up to I18c99dcd057435a15ebdf2faf55c7cbfc83cb47f that fixes an issue that lead to getMappingResultsForProperty() returning an empty result if no formObjectName was specified.

Related: FLOW-215

[!!!][BUGFIX] Submitted form data has precedence over value argument

This adjusts the behavior of all Form ViewHelpers so that any submitted value is redisplayed even if a “value” argument has been specified.

Being able to specify the “value” argument in Form ViewHelpers is a good way to pre-format the initial value:

<f:form.textfield property="price"
    value="{product.price -> f:format.number()}" />

The issue with this, however, was that upon re-display of the form due to property-mapping or validation errors the value argument had precedence over the previously submitted value.

This is a breaking change if you expect the previous behavior of form ViewHelpers always being pre-populated with the specified value attribute / bound object property even when re-displaying the form upon validation errors. Besides this change deprecates AbstractFormFieldViewHelper::getValue(). If you call that method in your custom ViewHelpers you should use AbstractFormFieldViewHelper::getValueAttribute() instead and call AbstractFormFieldViewHelper::addAdditionalIdentityPropertiesIfNeeded() explicitly if the ViewHelper might be bound to (sub)entities.

Fixes: FLOW-213

[BUGFIX] Highlight validation errors for unbound Form ViewHelpers

This change makes sure that the error class attribute is set on validation/property-mapping errors even if the “property” argument is not specified on the respective Form ViewHelper.

Fixes: FLOW-215

[BUGFIX] Fix risky unit tests

Adjusts two unit tests that are marked “risky”/fail when running PhpUnit in strict mode.

[!!!][FEATURE] Consistent escaping behavior

This is a major rework of the interceptors that are currently mostly used to automatically apply htmlspecialchars() to dynamic strings in Fluid templates.

This is a breaking change because it affects the basic escaping behavior of Fluid:

The escaping interceptor is now always enabled by default. Previously this was only the case if the request format was unknown or equal to “html”. To disable the automatic escaping add {escapingEnabled=false} anywhere in the template or (preferably) use the raw ViewHelper:

{objectAccess -> f:format.raw()}
{x:some.viewHelper() -> f:format.raw()}
{objectAccess -> x:some.viewHelper() -> f:format.raw()}
<f:format.raw><x:some.viewHelper /></f:format.raw>

Furthermore the escapingInterceptorEnabled flag in the AbstractViewHelper has been deprecated in favor of a new flag escapeChildren. The behavior of the flag is still the same though and the old name will still work.

Lastly the output of ViewHelpers is now also escaped by default! Previously ViewHelper authors had to take care of that themselves which was error-prone and less flexible. The escaping of a custom ViewHelper can be disabled by setting the new flag escapeOutput to FALSE in the ViewHelper class. But this should only be necessary if:

  1. The result of $this->renderChildren() is used directly as output (child nodes are escaped by default).
  2. The ViewHelper renders HTML code. Beware: In that case the output will need manual data sanitization ViewHelpers extending AbstractTagBasedViewHelper will already have the flag set.

All provided ViewHelpers are adjusted accordingly with one exception: The output of URI-ViewHelpers such as uri.action or widget.uri is now escaped for consistency reasons. If those are used to render HTML tag attributes the new behavior is desired because those will be properly encoded now. If the result of an URI ViewHelper is used directly, for example within some inline JavaScript the new escaping might break. In this case the raw ViewHelper can be used, as described above like done in the Index.html template of the Autocomplete widget.

Affected packages can be adjusted automatically by running provided core migration:

./flow core:migrate --version 20150214130800

Depends: If66a2dff21b239963728963f15437599a8442f72

Resolves: FLOW-26

[FEATURE] Allow usage of “else” argument with child nodes in AbstractConditionViewHelper

This feature allows for mixing usage of arguments and child node rendering for all AbstractConditionViewHelpers. This allows for writing:

<f:if condition="{products}" else="There are no products">
  <f:for each="{products}">
    <!-- ... -->
  </f:for>
</f:if>

To achieve what previously had to be written as:

<f:if condition="{products}">
  <f:then>
    <f:for each="{products}">
      <!-- ... -->
    </f:for>
  </f:then>
  <f:else>
    There are no products
  </f:else>
</f:if>

Resolves: FLOW-200

[BUGFIX] Render Form CSRF token field only if authenticated

Currently CSRF tokens are only enforced if an account is authenticated. But the form ViewHelper rendered the corresponding hidden field for all forms with method != “GET”.

Background:

Rendering the hidden field did not have a side effect before but as CSRF tokens only make sense with an active session, Security\\Context::getCsrfProtectionToken() will be adjusted to start a session when called. Therefore the token should only be fetched if it’s really required.

Related: FLOW-130

[FEATURE] “account” option for security.ifHasRole view helper

This change introduces a new option “account” for the ifHasRole view helper which allows for specifying an account other than the currently authenticated account.

Additionally the “role” option now allows roles to be specified as Role objects.

Resolves: FLOW-182

[TASK] Allow registering namespaces multiple times unless there are conflicts

Currently the TemplateParser throws an exception if the same namespace identifier is registered multiple times.

With this change this exception is skipped if the repeated registration does not pose conflicts, i.e. points to the same PHP namespace as previous mappings.

Background: This is a follow-up to I965cb54c3125f80e7a5ae46ede72ee9027ed006e that registers all package keys as Fluid namespaces. Due to some unrelated bug PackageManager::getActivePackages() can return the same package key twice leading to an exception: “#1224241246: Namespace identifier “xyz” is already registered.”.

Related: FLOW-151

[TASK] Add missing line breaks at the end of files

Most of TYPO3 Fluid’s PHP files end with a line break (empty line). This change adds this line break at the end of PHP files if it is missing.

This is not part of the CGL but increases consistency in this area.

This regular expression was used to add the empty lines:

Search:

(})\\Z(?!\\n)

Replace:

$1\\n

Fixes: FLOW-161

Revert “[!!!][FEATURE] Consistent escaping behavior”

This reverts commit ebc454f5b6d55a21bee940d0ab48e6dc534bf9b5 because that change breaks FE and BE rendering of Neos.

[!!!][FEATURE] Consistent escaping behavior

This is a major rework of the interceptors that are currently mostly used to automatically apply htmlspecialchars() to dynamic strings in Fluid templates.

This is a breaking change because it affects the basic escaping behavior of Fluid:

The escaping interceptor is now always enabled by default. Previously this was only the case if the request format was unknown or equal to “html”. To disable the automatic escaping add {escapingEnabled=false} anywhere in the template or (preferably) use the raw ViewHelper:

{objectAccess -> f:format.raw()}
{x:some.viewHelper() -> f:format.raw()}
{objectAccess -> x:some.viewHelper() -> f:format.raw()}
<f:format.raw><x:some.viewHelper /></f:format.raw>

Furthermore the escapingInterceptorEnabled flag in the AbstractViewHelper has been deprecated in favor of a new flag escapeChildren. The behavior of the flag is still the same though and the old name will still work.

Lastly the output of ViewHelpers is now also escaped by default! Previously ViewHelper authors had to take care of that themselves which was error-prone and less flexible. The escaping of a custom ViewHelper can be disabled by setting the new flag escapeOutput to FALSE in the ViewHelper class. But this should only be necessary if:

  1. The result of $this->renderChildren() is used directly as output (child nodes are escaped by default).
  2. The ViewHelper renders HTML code. Beware: In that case the output will need manual data sanitization ViewHelpers extending AbstractTagBasedViewHelper will already have the flag set.

All provided ViewHelpers are adjusted accordingly with one exception: The output of URI-ViewHelpers such as uri.action or widget.uri is now escaped for consistency reasons. If those are used to render HTML tag attributes the new behavior is desired because those will be properly encoded now. If the result of an URI ViewHelper is used directly, for example within some inline JavaScript the new escaping might break. In this case the raw ViewHelper can be used, as described above like done in the Index.html template of the Autocomplete widget.

Resolves: FLOW-26

[FEATURE] Configurable namespaces

This adds a signal to the TemplateParser that allows for registering ViewHelper namespaces.

With I965cb54c3125f80e7a5ae46ede72ee9027ed006e that is used to register all Flow keys of active packages as ViewHelper namespaces.

Resolves: FLOW-151

[!!!][FEATURE] Throw exception for unresolved namespaces

With this change the Fluid parser now throws an exception when it comes across an unknown ViewHelper namespace.

That is especially helpful if you forgot to import a namespace or mistyped a ViewHelper name.

It is a breaking change if you rely on the previous behavior of ignoring ViewHelpers with unknown namespaces. In that case you can ignore all unknown namespaces with:

{namespace *}

Specific namespaces can be ignored like this:

{namespace xs*}  <!-- ignores namespaces starting with "xs" -->
{namespace foo}  <!-- ignores the namespace "foo" -->

Resolves: FLOW-150

[TASK] Streamline regular expression for Resource URLs

The regular expression to split templates finding resource URLs is prone to breaking in case the match grows too long because it has no boundaries. This change stops the match earlier, this also massively increases the matching speed.

[BUGFIX] Throw helpful exception if ViewHelper class can’t be resolved

This adds some case sensitive checks for the resolved ViewHelper class name in order to prevent misleading fatal errors.

Background:

Previously, if the user mis-spelled a ViewHelper, e.g. by typing <f:format.textField> instead of <f:format.textfield>, the system failed with a fatal error of the following form:

Fatal error: Call to a member function getMethodParameters() on a non-object
in .../Core/ViewHelper/AbstractViewHelper.php on line 349

The cause of this error is that the reflection service was not properly injected into the ViewHelper, which happens because the class loader of Flow loads the non-instrumented class instead of the instrumented one.

This, in turn, happens because Data/Temporary/.../AvailableProxyClasses.php (which was introduced in Ie09b4e8891b61b33fd9bba3627a8312be02b1486) contains only correctly-spelled class names; making the classloader believe that it is not responsible for the ViewHelper.

Fixes: FLOW-152

[BUGFIX] Fix documentation (default vs. value) in TranslateViewHelper

The documentation in the header of TranslateViewHelper didn’t represent the new naming of attributes. The former “default” is now called “value”.

Resolves: FLOW-77

[TASK] Use Filesize utility functions in ByteViewHelper

Adjusts the format.bytes ViewHelper to use the Utility\\Files::bytesToSizeString() function introduced with I9b35d1b08c7cb1f41330d88f62fc1092e90880c6

Depends: I9b35d1b08c7cb1f41330d88f62fc1092e90880c6

[TASK] Improve ViewHelper documentation

The documentation of a few ViewHelpers is tweaked so it renders correctly.

[FEATURE] Provide option for resourced collection in upload view helper

This changes allows for specifying the name of the resource collection where the file should be uploaded to. If it is not specified, the default persistent resources collection will be used.

Related: NEOS-787

[TASK] Translate ViewHelper: Throw exception if current package can’t be resolved

Adjusts the f:translate ViewHelper to throw an exception if the package key can’t be determined from the current request.

Related: NEOS-497

[!!!] Remove usage of ReflectionService in ViewHelpers

The AbstractViewHelper now uses compile static to get all needed information about the render method of ViewHelper implementations.

As the AbstractViewHelper doesn’t use the ReflectionService anymore it was removed. This is breaking if a ViewHelper implementation relies on the fact that $this->reflectionService is available. A code migration warns about the possible usage of it.

[TASK] Adjustments for the new resource management

Related: FLOW-108

[TASK] Adjust to acl changes in Flow

Adjusts the security.ifAccess ViewHelper to the recent refactoring of the Security Framework in TYPO3.Flow. Namely the argument “resource” has been renamed to “privilegeTarget”.

This change also provides a code migration that should adjust affected Fluid templates. Make sure to run:

./flow core:migrate

in order to apply those changes.

Related: FLOW-11

[BUGFIX] Adjust code migration identifier pattern to contain the full timestamp

Previously code migrations are expected to have a class name with the pattern Version<YYYYMMDDhhmm> and the unique identifier was determined extracting the last 12 characters of the class name (which are expected to be the timestamp).

This change adjusts existing code migrations to use the full timestamp in order to establish the new guideline (note: those migrations still return the old identifier so that they won’t be applied again with a new identifier).

Related: FLOW-110

[TASK] Defer variable-initialization in AbstractViewHelper

Tweaks AbstractViewHelper::validateArguments() by deferring a variable initialization until it’s really used.

[BUGFIX] Hidden checkbox fields are disabled correctly

Until now checkboxes that were disabled would still have their empty hidden value be submitted, leading to checkbox values being overwritten.

This change correctly sets the hidden fields to the same disabled state as the related field by adjusting the form viewhelper variable ‘emptyHiddenFieldNames’.

Fixes: FLOW-255

[TASK] Remove unused method parameter

[BUGFIX] Stabilize “Consistent escaping behavior” core migration

This tweaks the TYPO3.Fluid-20150214130800 core migration to always include the ViewHelper file before adding the escapeOutput flag. Besides it now uses the PhpAnalyzer class to extract the ViewHelper class name.

Background: When executing core migrations the class loader might not be initialized yet. This makes sure the ViewHelpers can be adjusted anyways.

Related: FLOW-26

[BUGFIX] Preselect active options in property bound Select ViewHelper

Resolution of FLOW-213 introduced a regression that in some cases prevents the SelectViewHelper from pre-selecting property-bound values.

Background: The change I80e7e664e7a1fa41dc36bdf89e331086c6815f78 deprecated the method AbstractFormFieldViewHelper::getValue() in favor of a newly introduced method getValueAttribute(). That method always converts object values using the PersistenceManager. In case the object is not known to the PersistenceManager it thus returns NULL leading to the above issue when using the SelectViewHelper in “objectAccessorMode” with the multiple argument.

Related: FLOW-213

[TASK] Support for quoted keys in Fluid arrays

The syntax for Fluid arrays is based on the JSON syntax, but up to now it didn’t support quoted arrays, so:

{x:someViewHelper(arrayArgument: '{"foo": "bar"}')}

wouldn’t parse.

With this change the parser is a little less strict for arrays allowing the keys to be quoted with single or double quotes. It also allows to use square brackets and dots within quoted keys:

<x:someViewHelper arrayArgument="{'this[will]': 'work'}" />
{x:someViewhelper(arrayArgument: '{"and.so": "will.this"}'}

Resolves: FLOW-248

[BUGFIX] Fix support for “value” argument on the form.upload ViewHelper

When specifying the value argument on the form.upload ViewHelper the generated hidden field was named something like:

object[property][__identity][originallySubmittedResource][__identity]

since the adjustments to the new Resource Management (see Ic72a4ba780ebdef8b0940f3b9d20fa2c8840f834).

This change fixes the name to exclude the first “__identity”. Besides it fixes the behavior when specifying a “default resource” using the value argument:

  • If a form is re-displayed due to validation errors, any originally transmitted resource will survive the re-submission.
  • Otherwise, if a resource (or its identifier) is specified as “value” argument, that resource will be used by default
  • Finally, when in “object accessor mode”, the value of the bound property will be used as fallback.

Any newly uploaded file will of course overrule the hidden field as before.

Related: FLOW-214

[TASK] Translate labels in paginate widget

The previous / next labels in the paginate widget are replaced with translations.

[BUGFIX] Fix support for multi-part namespace identifiers

This change fixes support for namespaces containing dots. Previously the TemplateParser skipped those when used in “shorthand syntax” and something like:

{foo.bar:baz()}

was never replaced.

Background: This limitation is not new but it was easy to work around in previous versions. Since Flow package keys are registered as namespace identifiers (see FLOW-151) this issue became more urgent.

Fixes: FLOW-228

[BUGFIX] Always respect property binding in hidden fields

This is a follow up to the Submitted form data has precedence over value argument change (I80e7e664e7a1fa41dc36bdf89e331086c6815f78) that had the nasty side effect of breaking hidden fields bound to properties of a view model.

Before this fix, a template like:

<f:form.hidden property="someProperty" />

Would lead to a hidden field with an empty value attribute.

Background: The custom implementation of the getValueAttribute() in the HiddenViewHelper was missing the evaluation of getPropertyValue(). This is now fixed for all of the form ViewHelpers by making AbstractFormFieldViewHelper::getValueAttribute() more generic. The bug only really occurred for hidden fields though because checkbox and radio fields require the value argument to be specified explicitly.

Related: FLOW-213

[BUGFIX] Fix doc comment about string comparison in IfViewHelper

The inline documentation in the IfViewHelper states, that strings can not be compared directly but arrays should be used as workaround. This isn’t true anymore since direct string comparison works.

Fixes: FLOW-328

[BUGFIX] Only count items in ForViewHelper when iteration argument is set

When using the iteration argument, the ForViewHelper provides information about the current iteration, including the cycle and total. Therefore it needs to count all elements. The problem is, that this happens even if the iteration argument is not specified leading to performance issues especially when iterating over a (subset of a) lot of (remote) items.

This patch fixes this by adding a check for the iteration argument.

Fixes: FLOW-325

[BUGFIX] Fix support for mixed case namespace identifiers

This change adjusts the TemplateParser to support namespace aliases containing upper case characters:

{namespace someNamespace=Some\\Package\\ViewHelpers}
<someNamespace:someViewHelper />

Without this fix the above ViewHelper tag would not be resolved but rendered as is.

Background: This is a regression introduced with the stricter namespace resolution (see I9c63e7d0b116f9d1baabf09c6c18e97d8deb83fa).

Fixes: FLOW-290 Related: FLOW-150

[TASK] Detect and warn about usage of “format.json” and “uri.*” ViewHelpers

The “Consistent escaping behavior” change (FLOW-26) comes with a code migration that adjusts custom ViewHelpers to disable escaping in order to stay backwards compatible.

But the behavior of some core ViewHelpers has changed. Especially f:format.json and f:uri.* ViewHelpers now apply htmlspecialchars to their output for security and consistency reasons.

This change adds a warning for every HTML file that makes use of one of those ViewHelpers to give the developer a heads-up.

[TASK] Allow arbitrary characters in array/object keys

Allow arbitrary characters in array/object keys to support more possible keys, e.g. @action argument for the link view helper.

TYPO3.Kickstart

[TASK] Mark recent core migrations applied

This commit doesn’t contain any changes, it simply marks recent migrations applied so that:

./flow flow:core:migrate --status

won’t show any open migrations for this package.

[BUGFIX] Use propertyName instead of modelName for form labels

The NewTemplate used the modelName for each property as label instead of the propertyName, this commit fixes that to be in line with the EditTemplate.

Kudos for finding and fixing to Guido.

[FEATURE] Allow to kickstart documentation for a package

The new kickstart:documentation command allows to create the needed skeleton for Sphinx-based documentation: conf.py, Makefile and a basic index.rst file.

(cherry picked from commit 2c4dcd294d19fa5bd6a5ba7c33280a2300b165b9)

TYPO3.Welcome

[TASK] Undo requirements adjustment from CI job

The release CI job adjusted requirements dutifully, but the result was not as expected, because we humans missed some needed changes.