Fix bin/publish: copy docs.dist from project root

Fix bin/publish: use correct .env path for rspade_system
Fix bin/publish script: prevent grep exit code 1 from terminating script

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
root
2025-10-21 02:08:33 +00:00
commit f6fac6c4bc
79758 changed files with 10547827 additions and 0 deletions

21
vendor/spatie/backtrace/LICENSE.md vendored Executable file
View File

@@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) Spatie bvba <info@spatie.be>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

216
vendor/spatie/backtrace/README.md vendored Executable file
View File

@@ -0,0 +1,216 @@
# A better PHP backtrace
[![Latest Version on Packagist](https://img.shields.io/packagist/v/spatie/backtrace.svg?style=flat-square)](https://packagist.org/packages/spatie/backtrace)
![Tests](https://github.com/spatie/backtrace/workflows/Tests/badge.svg)
[![Total Downloads](https://img.shields.io/packagist/dt/spatie/backtrace.svg?style=flat-square)](https://packagist.org/packages/spatie/backtrace)
To get the backtrace in PHP you can use the `debug_backtrace` function. By default, it can be hard to work with. The
reported function name for a frame is skewed: it belongs to the previous frame. Also, options need to be passed using a bitmask.
This package provides a better way than `debug_backtrace` to work with a back trace. Here's an example:
```php
// returns an array with `Spatie\Backtrace\Frame` instances
$frames = Spatie\Backtrace\Backtrace::create()->frames();
$firstFrame = $frames[0];
$firstFrame->file; // returns the file name
$firstFrame->lineNumber; // returns the line number
$firstFrame->class; // returns the class name
```
## Support us
[<img src="https://github-ads.s3.eu-central-1.amazonaws.com/backtrace.jpg?t=1" width="419px" />](https://spatie.be/github-ad-click/backtrace)
We invest a lot of resources into creating [best in class open source packages](https://spatie.be/open-source). You can
support us by [buying one of our paid products](https://spatie.be/open-source/support-us).
We highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using.
You'll find our address on [our contact page](https://spatie.be/about-us). We publish all received postcards
on [our virtual postcard wall](https://spatie.be/open-source/postcards).
## Installation
You can install the package via composer:
```bash
composer require spatie/backtrace
```
## Usage
This is how you can create a backtrace instance:
```php
$backtrace = Spatie\Backtrace\Backtrace::create();
```
### Getting the frames
To get all the frames you can call `frames`.
```php
$frames = $backtrace->frames(); // contains an array with `Spatie\Backtrace\Frame` instances
```
A `Spatie\Backtrace\Frame` has these properties:
- `file`: the name of the file
- `lineNumber`: the line number
- `arguments`: the arguments used for this frame. Will be `null` if `withArguments` was not used.
- `class`: the class name for this frame. Will be `null` if the frame concerns a function.
- `method`: the method used in this frame
- `object`: the object when the frame is in an object context (method call, closure bound to object, arrow function which captured `$this`, etc.). Will be `null` if `withObject` was not used.
- `applicationFrame`: contains `true` is this frame belongs to your application, and `false` if it belongs to a file in
the vendor directory. A couple edge cases exist, see "application frames" below.
### Collecting arguments and objects
For performance reasons, the frames of the back trace will not contain the arguments of the called functions and the
object. If you want to add those, use the `withArguments` and `withObject` methods.
```php
$backtrace = Spatie\Backtrace\Backtrace::create()->withArguments()->withObject();
```
#### Reducing arguments
For viewing purposes, arguments can be reduced to a string:
```php
$backtrace = Spatie\Backtrace\Backtrace::create()->withArguments()->reduceArguments();
```
By default, some typical types will be reduced to a string. You can define your own reduction algorithm per type by implementing an `ArgumentReducer`:
```php
class DateTimeWithOtherFormatArgumentReducer implements ArgumentReducer
{
public function execute($argument): ReducedArgumentContract
{
if (! $argument instanceof DateTimeInterface) {
return UnReducedArgument::create();
}
return new ReducedArgument(
$argument->format('d/m/y H:i'),
get_class($argument),
);
}
}
```
This is a copy of the built-in argument reducer for `DateTimeInterface` where we've updated the format. An `UnReducedArgument` object is returned when the argument is not of the expected type. A `ReducedArgument` object is returned with the reduced value of the argument and the original type of the argument.
The reducer can be used as such:
```php
$backtrace = Spatie\Backtrace\Backtrace::create()->withArguments()->reduceArguments(
Spatie\Backtrace\Arguments\ArgumentReducers::default([
new DateTimeWithOtherFormatArgumentReducer()
])
);
```
Which will first execute the new reducer and then the default ones.
### Setting the application path
You can use the `applicationPath` to pass the base path of your app. This value will be used to determine whether a
frame is an application frame, or a vendor frame. Here's an example using a Laravel specific function.
```php
$backtrace = Spatie\Backtrace\Backtrace::create()->applicationPath(base_path());
```
### Removing the application path from the file name
You can use `trimFilePaths` to remove the base path of your app from the file. This will only work if you use it in conjunction with the `applicationPath` method re above. Here's an example using a Laravel specific function. This will ensure the Frame has the trimmedFilePath property set.
```php
$backtrace = Backtrace::create()->applicationPath(base_path())->trimFilePaths());
```
### Application frames
By default, a frame is considered an application frame when the file of the frame is not in the vendor directory. The following edge cases apply:
- Laravel's `artisan` CLI file (generally found in the root of the project) is considered a vendor file and frame
- Statamic's `please` CLI file (generally found in the root of the project) is considered a vendor file and frame
### Getting a certain part of a trace
If you only want to have the frames starting from a particular frame in the backtrace you can use
the `startingFromFrame` method:
```php
use Spatie\Backtrace\Backtrace;
use Spatie\Backtrace\Frame;
$frames = Backtrace::create()
->startingFromFrame(function (Frame $frame) {
return $frame->class === MyClass::class;
})
->frames();
```
With this code, all frames before the frame that concerns `MyClass` will have been filtered out.
Alternatively, you can use the `offset` method, which will skip the given number of frames. In this example the first 2 frames will not end up in `$frames`.
```php
$frames = Spatie\Backtrace\Backtrace::create()
->offset(2)
->frames();
```
### Limiting the number of frames
To only get a specific number of frames use the `limit` function. In this example, we'll only get the first two frames.
```php
$frames = Spatie\Backtrace\Backtrace::create()
->limit(2)
->frames();
```
### Getting a backtrace for a throwable
Here's how you can get a backtrace for a throwable.
```php
$frames = Spatie\Backtrace\Backtrace::createForThrowable($throwable)
```
Because we will use the backtrace that is already available in the throwable, the frames will contain the arguments used
in the backtrace as long as the `zend.exception_ignore_args` INI option is disabled (set to `0`) *before* the throwable
is thrown. On the other hand, objects will never be included in the backtrace.
[More information](https://www.php.net/manual/en/throwable.gettrace.php#129087).
## Testing
``` bash
composer test
```
## Changelog
Please see [CHANGELOG](CHANGELOG.md) for more information on what has changed recently.
## Contributing
Please see [CONTRIBUTING](https://github.com/spatie/.github/blob/main/CONTRIBUTING.md) for details.
## Security Vulnerabilities
Please review [our security policy](../../security/policy) on how to report security vulnerabilities.
## Credits
- [Freek Van de Herten](https://github.com/freekmurze)
- [All Contributors](../../contributors)
## License
The MIT License (MIT). Please see [License File](LICENSE.md) for more information.

59
vendor/spatie/backtrace/composer.json vendored Executable file
View File

@@ -0,0 +1,59 @@
{
"name": "spatie/backtrace",
"description": "A better backtrace",
"license": "MIT",
"keywords": [
"spatie",
"backtrace"
],
"authors": [
{
"name": "Freek Van de Herten",
"email": "freek@spatie.be",
"homepage": "https://spatie.be",
"role": "Developer"
}
],
"homepage": "https://github.com/spatie/backtrace",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/spatie"
},
{
"type": "other",
"url": "https://spatie.be/open-source/support-us"
}
],
"require": {
"php": "^7.3 || ^8.0"
},
"require-dev": {
"ext-json": "*",
"laravel/serializable-closure": "^1.3 || ^2.0",
"phpunit/phpunit": "^9.3 || ^11.4.3",
"spatie/phpunit-snapshot-assertions": "^4.2 || ^5.1.6",
"symfony/var-dumper": "^5.1 || ^6.0 || ^7.0"
},
"minimum-stability": "dev",
"prefer-stable": true,
"autoload": {
"psr-4": {
"Spatie\\Backtrace\\": "src"
}
},
"autoload-dev": {
"psr-4": {
"Spatie\\Backtrace\\Tests\\": "tests"
}
},
"config": {
"sort-packages": true
},
"scripts": {
"format": "vendor/bin/php-cs-fixer fix --allow-risky=yes",
"psalm": "vendor/bin/psalm",
"test": "vendor/bin/phpunit",
"test-coverage": "vendor/bin/phpunit --coverage-html coverage"
}
}

View File

@@ -0,0 +1,81 @@
<?php
namespace Spatie\Backtrace\Arguments;
use Spatie\Backtrace\Arguments\Reducers\ArgumentReducer;
use Spatie\Backtrace\Arguments\Reducers\ArrayArgumentReducer;
use Spatie\Backtrace\Arguments\Reducers\BaseTypeArgumentReducer;
use Spatie\Backtrace\Arguments\Reducers\ClosureArgumentReducer;
use Spatie\Backtrace\Arguments\Reducers\DateTimeArgumentReducer;
use Spatie\Backtrace\Arguments\Reducers\DateTimeZoneArgumentReducer;
use Spatie\Backtrace\Arguments\Reducers\EnumArgumentReducer;
use Spatie\Backtrace\Arguments\Reducers\MinimalArrayArgumentReducer;
use Spatie\Backtrace\Arguments\Reducers\SensitiveParameterArrayReducer;
use Spatie\Backtrace\Arguments\Reducers\StdClassArgumentReducer;
use Spatie\Backtrace\Arguments\Reducers\StringableArgumentReducer;
use Spatie\Backtrace\Arguments\Reducers\SymphonyRequestArgumentReducer;
class ArgumentReducers
{
/** @var array<int, ArgumentReducer> */
public $argumentReducers = [];
/**
* @param array<ArgumentReducer|class-string<ArgumentReducer>> $argumentReducers
*/
public static function create(array $argumentReducers): self
{
return new self(array_map(
function ($argumentReducer) {
/** @var $argumentReducer ArgumentReducer|class-string<ArgumentReducer> */
return $argumentReducer instanceof ArgumentReducer ? $argumentReducer : new $argumentReducer();
},
$argumentReducers
));
}
public static function default(array $extra = []): self
{
return new self(static::defaultReducers($extra));
}
public static function minimal(array $extra = []): self
{
return new self(static::minimalReducers($extra));
}
/**
* @param array<int, ArgumentReducer> $argumentReducers
*/
protected function __construct(array $argumentReducers)
{
$this->argumentReducers = $argumentReducers;
}
protected static function defaultReducers(array $extra = []): array
{
return array_merge($extra, [
new BaseTypeArgumentReducer(),
new ArrayArgumentReducer(),
new StdClassArgumentReducer(),
new EnumArgumentReducer(),
new ClosureArgumentReducer(),
new SensitiveParameterArrayReducer(),
new DateTimeArgumentReducer(),
new DateTimeZoneArgumentReducer(),
new SymphonyRequestArgumentReducer(),
new StringableArgumentReducer(),
]);
}
protected static function minimalReducers(array $extra = []): array
{
return array_merge($extra, [
new BaseTypeArgumentReducer(),
new MinimalArrayArgumentReducer(),
new EnumArgumentReducer(),
new ClosureArgumentReducer(),
new SensitiveParameterArrayReducer(),
]);
}
}

View File

@@ -0,0 +1,118 @@
<?php
namespace Spatie\Backtrace\Arguments;
use ReflectionParameter;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgument;
use Spatie\Backtrace\Arguments\ReducedArgument\TruncatedReducedArgument;
class ProvidedArgument
{
/** @var string */
public $name;
/** @var bool */
public $passedByReference = false;
/** @var bool */
public $isVariadic = false;
/** @var bool */
public $hasDefaultValue = false;
/** @var mixed */
public $defaultValue = null;
/** @var bool */
public $defaultValueUsed = false;
/** @var bool */
public $truncated = false;
/** @var mixed */
public $reducedValue = null;
/** @var string|null */
public $originalType = null;
public static function fromReflectionParameter(ReflectionParameter $parameter): self
{
return new self(
$parameter->getName(),
$parameter->isPassedByReference(),
$parameter->isVariadic(),
$parameter->isDefaultValueAvailable(),
$parameter->isDefaultValueAvailable() ? $parameter->getDefaultValue() : null,
);
}
public static function fromNonReflectableParameter(
int $index
): self {
return new self(
"arg{$index}",
false,
);
}
public function __construct(
string $name,
bool $passedByReference = false,
bool $isVariadic = false,
bool $hasDefaultValue = false,
$defaultValue = null,
bool $defaultValueUsed = false,
bool $truncated = false,
$reducedValue = null,
?string $originalType = null
) {
$this->originalType = $originalType;
$this->reducedValue = $reducedValue;
$this->truncated = $truncated;
$this->defaultValueUsed = $defaultValueUsed;
$this->defaultValue = $defaultValue;
$this->hasDefaultValue = $hasDefaultValue;
$this->isVariadic = $isVariadic;
$this->passedByReference = $passedByReference;
$this->name = $name;
if ($this->isVariadic) {
$this->defaultValue = [];
}
}
public function setReducedArgument(
ReducedArgument $reducedArgument
): self {
$this->reducedValue = $reducedArgument->value;
$this->originalType = $reducedArgument->originalType;
if ($reducedArgument instanceof TruncatedReducedArgument) {
$this->truncated = true;
}
return $this;
}
public function defaultValueUsed(): self
{
$this->defaultValueUsed = true;
$this->originalType = get_debug_type($this->defaultValue);
return $this;
}
public function toArray(): array
{
return [
'name' => $this->name,
'value' => $this->defaultValueUsed
? $this->defaultValue
: $this->reducedValue,
'original_type' => $this->originalType,
'passed_by_reference' => $this->passedByReference,
'is_variadic' => $this->isVariadic,
'truncated' => $this->truncated,
];
}
}

View File

@@ -0,0 +1,44 @@
<?php
namespace Spatie\Backtrace\Arguments;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgument;
class ReduceArgumentPayloadAction
{
/** @var \Spatie\Backtrace\Arguments\ArgumentReducers */
protected $argumentReducers;
public function __construct(
ArgumentReducers $argumentReducers
) {
$this->argumentReducers = $argumentReducers;
}
public function reduce($argument, bool $includeObjectType = false): ReducedArgument
{
foreach ($this->argumentReducers->argumentReducers as $reducer) {
$reduced = $reducer->execute($argument);
if ($reduced instanceof ReducedArgument) {
return $reduced;
}
}
if (gettype($argument) === 'object' && $includeObjectType) {
return new ReducedArgument(
'object ('.get_class($argument).')',
get_debug_type($argument),
);
}
if (gettype($argument) === 'object') {
return new ReducedArgument('object', get_debug_type($argument), );
}
return new ReducedArgument(
$argument,
get_debug_type($argument),
);
}
}

View File

@@ -0,0 +1,117 @@
<?php
namespace Spatie\Backtrace\Arguments;
use ReflectionException;
use ReflectionFunction;
use ReflectionMethod;
use ReflectionParameter;
use Spatie\Backtrace\Arguments\ReducedArgument\VariadicReducedArgument;
use Throwable;
class ReduceArgumentsAction
{
/** @var ArgumentReducers */
protected $argumentReducers;
/** @var ReduceArgumentPayloadAction */
protected $reduceArgumentPayloadAction;
public function __construct(
ArgumentReducers $argumentReducers
) {
$this->argumentReducers = $argumentReducers;
$this->reduceArgumentPayloadAction = new ReduceArgumentPayloadAction($argumentReducers);
}
public function execute(
?string $class,
?string $method,
?array $frameArguments
): ?array {
try {
if ($frameArguments === null) {
return null;
}
$parameters = $this->getParameters($class, $method);
if ($parameters === null) {
$arguments = [];
foreach ($frameArguments as $index => $argument) {
$arguments[$index] = ProvidedArgument::fromNonReflectableParameter($index)
->setReducedArgument($this->reduceArgumentPayloadAction->reduce($argument))
->toArray();
}
return $arguments;
}
$arguments = array_map(
function ($argument) {
return $this->reduceArgumentPayloadAction->reduce($argument);
},
$frameArguments,
);
$argumentsCount = count($arguments);
$hasVariadicParameter = false;
foreach ($parameters as $index => $parameter) {
if ($index + 1 > $argumentsCount) {
$parameter->defaultValueUsed();
} elseif ($parameter->isVariadic) {
$parameter->setReducedArgument(new VariadicReducedArgument(array_slice($arguments, $index)));
$hasVariadicParameter = true;
} else {
$parameter->setReducedArgument($arguments[$index]);
}
$parameters[$index] = $parameter->toArray();
}
if ($this->moreArgumentsProvidedThanParameters($arguments, $parameters, $hasVariadicParameter)) {
for ($i = count($parameters); $i < count($arguments); $i++) {
$parameters[$i] = ProvidedArgument::fromNonReflectableParameter(count($parameters))
->setReducedArgument($arguments[$i])
->toArray();
}
}
return $parameters;
} catch (Throwable $e) {
return null;
}
}
/** @return null|Array<\Spatie\Backtrace\Arguments\ProvidedArgument> */
protected function getParameters(
?string $class,
?string $method
): ?array {
try {
$reflection = $class !== null
? new ReflectionMethod($class, $method)
: new ReflectionFunction($method);
} catch (ReflectionException $e) {
return null;
}
return array_map(
function (ReflectionParameter $reflectionParameter) {
return ProvidedArgument::fromReflectionParameter($reflectionParameter);
},
$reflection->getParameters(),
);
}
protected function moreArgumentsProvidedThanParameters(
array $arguments,
array $parameters,
bool $hasVariadicParameter
): bool {
return count($arguments) > count($parameters) && ! $hasVariadicParameter;
}
}

View File

@@ -0,0 +1,23 @@
<?php
namespace Spatie\Backtrace\Arguments\ReducedArgument;
class ReducedArgument implements ReducedArgumentContract
{
/** @var mixed */
public $value;
/** @var string */
public $originalType;
/**
* @param mixed $value
*/
public function __construct(
$value,
string $originalType
) {
$this->originalType = $originalType;
$this->value = $value;
}
}

View File

@@ -0,0 +1,8 @@
<?php
namespace Spatie\Backtrace\Arguments\ReducedArgument;
interface ReducedArgumentContract
{
}

View File

@@ -0,0 +1,8 @@
<?php
namespace Spatie\Backtrace\Arguments\ReducedArgument;
class TruncatedReducedArgument extends ReducedArgument
{
}

View File

@@ -0,0 +1,22 @@
<?php
namespace Spatie\Backtrace\Arguments\ReducedArgument;
class UnReducedArgument implements ReducedArgumentContract
{
/** @var self|null */
private static $instance = null;
private function __construct()
{
}
public static function create(): self
{
if (self::$instance !== null) {
return self::$instance;
}
return self::$instance = new self();
}
}

View File

@@ -0,0 +1,21 @@
<?php
namespace Spatie\Backtrace\Arguments\ReducedArgument;
use Exception;
class VariadicReducedArgument extends ReducedArgument
{
public function __construct(array $value)
{
foreach ($value as $key => $item) {
if (! $item instanceof ReducedArgument) {
throw new Exception('VariadicReducedArgument must be an array of ReducedArgument');
}
$value[$key] = $item->value;
}
parent::__construct($value, 'array');
}
}

View File

@@ -0,0 +1,13 @@
<?php
namespace Spatie\Backtrace\Arguments\Reducers;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgumentContract;
interface ArgumentReducer
{
/**
* @param mixed $argument
*/
public function execute($argument): ReducedArgumentContract;
}

View File

@@ -0,0 +1,52 @@
<?php
namespace Spatie\Backtrace\Arguments\Reducers;
use Spatie\Backtrace\Arguments\ArgumentReducers;
use Spatie\Backtrace\Arguments\ReduceArgumentPayloadAction;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgument;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgumentContract;
use Spatie\Backtrace\Arguments\ReducedArgument\TruncatedReducedArgument;
use Spatie\Backtrace\Arguments\ReducedArgument\UnReducedArgument;
class ArrayArgumentReducer implements ReducedArgumentContract
{
/** @var int */
protected $maxArraySize = 25;
/** @var \Spatie\Backtrace\Arguments\ReduceArgumentPayloadAction */
protected $reduceArgumentPayloadAction;
public function __construct()
{
$this->reduceArgumentPayloadAction = new ReduceArgumentPayloadAction(ArgumentReducers::minimal());
}
public function execute($argument): ReducedArgumentContract
{
if (! is_array($argument)) {
return UnReducedArgument::create();
}
return $this->reduceArgument($argument, 'array');
}
protected function reduceArgument(array $argument, string $originalType): ReducedArgumentContract
{
foreach ($argument as $key => $value) {
$argument[$key] = $this->reduceArgumentPayloadAction->reduce(
$value,
true
)->value;
}
if (count($argument) > $this->maxArraySize) {
return new TruncatedReducedArgument(
array_slice($argument, 0, $this->maxArraySize),
'array'
);
}
return new ReducedArgument($argument, $originalType);
}
}

View File

@@ -0,0 +1,24 @@
<?php
namespace Spatie\Backtrace\Arguments\Reducers;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgument;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgumentContract;
use Spatie\Backtrace\Arguments\ReducedArgument\UnReducedArgument;
class BaseTypeArgumentReducer implements ArgumentReducer
{
public function execute($argument): ReducedArgumentContract
{
if (is_int($argument)
|| is_float($argument)
|| is_bool($argument)
|| is_string($argument)
|| $argument === null
) {
return new ReducedArgument($argument, get_debug_type($argument));
}
return UnReducedArgument::create();
}
}

View File

@@ -0,0 +1,30 @@
<?php
namespace Spatie\Backtrace\Arguments\Reducers;
use Closure;
use ReflectionFunction;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgument;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgumentContract;
use Spatie\Backtrace\Arguments\ReducedArgument\UnReducedArgument;
class ClosureArgumentReducer implements ArgumentReducer
{
public function execute($argument): ReducedArgumentContract
{
if (! $argument instanceof Closure) {
return UnReducedArgument::create();
}
$reflection = new ReflectionFunction($argument);
if ($reflection->getFileName() && $reflection->getStartLine() && $reflection->getEndLine()) {
return new ReducedArgument(
"{$reflection->getFileName()}:{$reflection->getStartLine()}-{$reflection->getEndLine()}",
'Closure'
);
}
return new ReducedArgument("{$reflection->getFileName()}", 'Closure');
}
}

View File

@@ -0,0 +1,23 @@
<?php
namespace Spatie\Backtrace\Arguments\Reducers;
use DateTimeInterface;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgument;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgumentContract;
use Spatie\Backtrace\Arguments\ReducedArgument\UnReducedArgument;
class DateTimeArgumentReducer implements ArgumentReducer
{
public function execute($argument): ReducedArgumentContract
{
if (! $argument instanceof DateTimeInterface) {
return UnReducedArgument::create();
}
return new ReducedArgument(
$argument->format('d M Y H:i:s e'),
get_class($argument),
);
}
}

View File

@@ -0,0 +1,23 @@
<?php
namespace Spatie\Backtrace\Arguments\Reducers;
use DateTimeZone;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgument;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgumentContract;
use Spatie\Backtrace\Arguments\ReducedArgument\UnReducedArgument;
class DateTimeZoneArgumentReducer implements ArgumentReducer
{
public function execute($argument): ReducedArgumentContract
{
if (! $argument instanceof DateTimeZone) {
return UnReducedArgument::create();
}
return new ReducedArgument(
$argument->getName(),
get_class($argument),
);
}
}

View File

@@ -0,0 +1,23 @@
<?php
namespace Spatie\Backtrace\Arguments\Reducers;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgument;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgumentContract;
use Spatie\Backtrace\Arguments\ReducedArgument\UnReducedArgument;
use UnitEnum;
class EnumArgumentReducer implements ArgumentReducer
{
public function execute($argument): ReducedArgumentContract
{
if (! $argument instanceof UnitEnum) {
return UnReducedArgument::create();
}
return new ReducedArgument(
get_class($argument).'::'.$argument->name,
get_class($argument),
);
}
}

View File

@@ -0,0 +1,22 @@
<?php
namespace Spatie\Backtrace\Arguments\Reducers;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgument;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgumentContract;
use Spatie\Backtrace\Arguments\ReducedArgument\UnReducedArgument;
class MinimalArrayArgumentReducer implements ArgumentReducer
{
public function execute($argument): ReducedArgumentContract
{
if (! is_array($argument)) {
return UnReducedArgument::create();
}
return new ReducedArgument(
'array (size='.count($argument).')',
'array'
);
}
}

View File

@@ -0,0 +1,23 @@
<?php
namespace Spatie\Backtrace\Arguments\Reducers;
use SensitiveParameterValue;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgument;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgumentContract;
use Spatie\Backtrace\Arguments\ReducedArgument\UnReducedArgument;
class SensitiveParameterArrayReducer implements ArgumentReducer
{
public function execute($argument): ReducedArgumentContract
{
if (! $argument instanceof SensitiveParameterValue) {
return UnReducedArgument::create();
}
return new ReducedArgument(
'SensitiveParameterValue('.get_debug_type($argument->getValue()).')',
get_class($argument)
);
}
}

View File

@@ -0,0 +1,19 @@
<?php
namespace Spatie\Backtrace\Arguments\Reducers;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgumentContract;
use Spatie\Backtrace\Arguments\ReducedArgument\UnReducedArgument;
use stdClass;
class StdClassArgumentReducer extends ArrayArgumentReducer
{
public function execute($argument): ReducedArgumentContract
{
if (! $argument instanceof stdClass) {
return UnReducedArgument::create();
}
return parent::reduceArgument((array) $argument, stdClass::class);
}
}

View File

@@ -0,0 +1,23 @@
<?php
namespace Spatie\Backtrace\Arguments\Reducers;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgument;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgumentContract;
use Spatie\Backtrace\Arguments\ReducedArgument\UnReducedArgument;
use Stringable;
class StringableArgumentReducer implements ArgumentReducer
{
public function execute($argument): ReducedArgumentContract
{
if (! $argument instanceof Stringable) {
return UnReducedArgument::create();
}
return new ReducedArgument(
(string) $argument,
get_class($argument),
);
}
}

View File

@@ -0,0 +1,23 @@
<?php
namespace Spatie\Backtrace\Arguments\Reducers;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgument;
use Spatie\Backtrace\Arguments\ReducedArgument\ReducedArgumentContract;
use Spatie\Backtrace\Arguments\ReducedArgument\UnReducedArgument;
use Symfony\Component\HttpFoundation\Request;
class SymphonyRequestArgumentReducer implements ArgumentReducer
{
public function execute($argument): ReducedArgumentContract
{
if (! $argument instanceof Request) {
return UnReducedArgument::create();
}
return new ReducedArgument(
"{$argument->getMethod()} {$argument->getUri()}",
get_class($argument),
);
}
}

312
vendor/spatie/backtrace/src/Backtrace.php vendored Executable file
View File

@@ -0,0 +1,312 @@
<?php
namespace Spatie\Backtrace;
use Closure;
use Laravel\SerializableClosure\Support\ClosureStream;
use Spatie\Backtrace\Arguments\ArgumentReducers;
use Spatie\Backtrace\Arguments\ReduceArgumentsAction;
use Spatie\Backtrace\Arguments\Reducers\ArgumentReducer;
use Throwable;
class Backtrace
{
/** @var bool */
protected $withArguments = false;
/** @var bool */
protected $reduceArguments = false;
/** @var array<class-string<ArgumentReducer>|ArgumentReducer>|ArgumentReducers|null */
protected $argumentReducers = null;
/** @var bool */
protected $withObject = false;
/** @var bool */
protected $trimFilePaths = false;
/** @var string|null */
protected $applicationPath;
/** @var int */
protected $offset = 0;
/** @var int */
protected $limit = 0;
/** @var \Closure|null */
protected $startingFromFrameClosure = null;
/** @var \Throwable|null */
protected $throwable = null;
public static function create(): self
{
return new static();
}
public static function createForThrowable(Throwable $throwable): self
{
return (new static())->forThrowable($throwable);
}
protected function forThrowable(Throwable $throwable): self
{
$this->throwable = $throwable;
return $this;
}
public function withArguments(
bool $withArguments = true
): self {
$this->withArguments = $withArguments;
return $this;
}
/**
* @param array<class-string<ArgumentReducer>|ArgumentReducer>|ArgumentReducers|null $argumentReducers
*
* @return $this
*/
public function reduceArguments(
$argumentReducers = null
): self {
$this->reduceArguments = true;
$this->argumentReducers = $argumentReducers;
return $this;
}
public function withObject(bool $withObject = true): self
{
$this->withObject = $withObject;
return $this;
}
public function applicationPath(string $applicationPath): self
{
$this->applicationPath = rtrim($applicationPath, '/');
return $this;
}
public function trimFilePaths(): self
{
$this->trimFilePaths = true;
return $this;
}
public function offset(int $offset): self
{
$this->offset = $offset;
return $this;
}
public function limit(int $limit): self
{
$this->limit = $limit;
return $this;
}
public function startingFromFrame(Closure $startingFromFrameClosure)
{
$this->startingFromFrameClosure = $startingFromFrameClosure;
return $this;
}
/**
* @return \Spatie\Backtrace\Frame[]
*/
public function frames(): array
{
$rawFrames = $this->getRawFrames();
return $this->toFrameObjects($rawFrames);
}
public function firstApplicationFrameIndex(): ?int
{
foreach ($this->frames() as $index => $frame) {
if ($frame->applicationFrame) {
return $index;
}
}
return null;
}
protected function getRawFrames(): array
{
if ($this->throwable) {
return $this->throwable->getTrace();
}
// Omit arguments and object
$options = DEBUG_BACKTRACE_IGNORE_ARGS;
// Populate arguments
if ($this->withArguments) {
$options = 0;
}
// Populate object
if ($this->withObject) {
$options = $options | DEBUG_BACKTRACE_PROVIDE_OBJECT;
}
$limit = $this->limit;
if ($limit !== 0) {
$limit += 3;
}
return debug_backtrace($options, $limit);
}
/**
* @return \Spatie\Backtrace\Frame[]
*/
protected function toFrameObjects(array $rawFrames): array
{
$currentFile = $this->throwable ? $this->throwable->getFile() : '';
$currentLine = $this->throwable ? $this->throwable->getLine() : 0;
$arguments = $this->withArguments ? [] : null;
$frames = [];
$reduceArgumentsAction = new ReduceArgumentsAction($this->resolveArgumentReducers());
foreach ($rawFrames as $rawFrame) {
$textSnippet = null;
if (
class_exists(ClosureStream::class)
&& substr($currentFile, 0, strlen(ClosureStream::STREAM_PROTO)) === ClosureStream::STREAM_PROTO
) {
$textSnippet = $currentFile;
$currentFile = ClosureStream::STREAM_PROTO.'://function()';
$currentLine -= 1;
}
if ($this->trimFilePaths && $this->applicationPath) {
$trimmedFilePath = str_replace($this->applicationPath, '', $currentFile);
}
$frame = new Frame(
$currentFile,
$currentLine,
$arguments,
$rawFrame['function'] ?? null,
$rawFrame['class'] ?? null,
$rawFrame['object'] ?? null,
$this->isApplicationFrame($currentFile),
$textSnippet,
$trimmedFilePath ?? null,
);
$frames[] = $frame;
$arguments = $this->withArguments
? $rawFrame['args'] ?? null
: null;
if ($this->reduceArguments) {
$arguments = $reduceArgumentsAction->execute(
$rawFrame['class'] ?? null,
$rawFrame['function'] ?? null,
$arguments
);
}
$currentFile = $rawFrame['file'] ?? 'unknown';
$currentLine = $rawFrame['line'] ?? 0;
}
$frames[] = new Frame(
$currentFile,
$currentLine,
[],
'[top]',
null,
null,
$this->isApplicationFrame($currentFile),
);
$frames = $this->removeBacktracePackageFrames($frames);
if ($closure = $this->startingFromFrameClosure) {
$frames = $this->startAtFrameFromClosure($frames, $closure);
}
$frames = array_slice($frames, $this->offset, $this->limit === 0 ? PHP_INT_MAX : $this->limit);
return array_values($frames);
}
protected function isApplicationFrame(string $frameFilename): bool
{
$relativeFile = str_replace('\\', DIRECTORY_SEPARATOR, $frameFilename);
if (! empty($this->applicationPath)) {
$relativeFile = array_reverse(explode($this->applicationPath ?? '', $frameFilename, 2))[0];
}
if (strpos($relativeFile, DIRECTORY_SEPARATOR.'vendor') === 0) {
return false;
}
// Edge case for vendor files that typically live in the app code (e.g. Laravel's `artisan` or Statamic's `please`)
if (preg_match('/\/(artisan|please)$/', $relativeFile)) {
return false;
}
return true;
}
protected function removeBacktracePackageFrames(array $frames): array
{
return $this->startAtFrameFromClosure($frames, function (Frame $frame) {
return $frame->class !== static::class;
});
}
/**
* @param \Spatie\Backtrace\Frame[] $frames
* @param \Closure $closure
*
* @return array
*/
protected function startAtFrameFromClosure(array $frames, Closure $closure): array
{
foreach ($frames as $i => $frame) {
$foundStartingFrame = $closure($frame);
if ($foundStartingFrame) {
return $frames;
}
unset($frames[$i]);
}
return $frames;
}
protected function resolveArgumentReducers(): ArgumentReducers
{
if ($this->argumentReducers === null) {
return ArgumentReducers::default();
}
if ($this->argumentReducers instanceof ArgumentReducers) {
return $this->argumentReducers;
}
return ArgumentReducers::create($this->argumentReducers);
}
}

View File

@@ -0,0 +1,77 @@
<?php
namespace Spatie\Backtrace\CodeSnippets;
use RuntimeException;
class CodeSnippet
{
/** @var int */
protected $surroundingLine = 1;
/** @var int */
protected $snippetLineCount = 9;
public function surroundingLine(int $surroundingLine): self
{
$this->surroundingLine = $surroundingLine;
return $this;
}
public function snippetLineCount(int $snippetLineCount): self
{
$this->snippetLineCount = $snippetLineCount;
return $this;
}
public function get(SnippetProvider $provider): array
{
try {
[$startLineNumber, $endLineNumber] = $this->getBounds($provider->numberOfLines());
$code = [];
$line = $provider->getLine($startLineNumber);
$currentLineNumber = $startLineNumber;
while ($currentLineNumber <= $endLineNumber) {
$code[$currentLineNumber] = rtrim(substr($line, 0, 250));
$line = $provider->getNextLine();
$currentLineNumber++;
}
return $code;
} catch (RuntimeException $exception) {
return [];
}
}
public function getAsString(SnippetProvider $provider): string
{
$snippet = $this->get($provider);
$snippetStrings = array_map(function (string $line, string $number) {
return "{$number} {$line}";
}, $snippet, array_keys($snippet));
return implode(PHP_EOL, $snippetStrings);
}
protected function getBounds(int $totalNumberOfLineInFile): array
{
$startLine = max($this->surroundingLine - floor($this->snippetLineCount / 2), 1);
$endLine = $startLine + ($this->snippetLineCount - 1);
if ($endLine > $totalNumberOfLineInFile) {
$endLine = $totalNumberOfLineInFile;
$startLine = max($endLine - ($this->snippetLineCount - 1), 1);
}
return [$startLine, $endLine];
}
}

View File

@@ -0,0 +1,41 @@
<?php
namespace Spatie\Backtrace\CodeSnippets;
use SplFileObject;
class FileSnippetProvider implements SnippetProvider
{
/** @var \SplFileObject */
protected $file;
public function __construct(string $path)
{
$this->file = new SplFileObject($path);
}
public function numberOfLines(): int
{
$this->file->seek(PHP_INT_MAX);
return $this->file->key() + 1;
}
public function getLine(?int $lineNumber = null): string
{
if (is_null($lineNumber)) {
return $this->getNextLine();
}
$this->file->seek($lineNumber - 1);
return $this->file->current();
}
public function getNextLine(): string
{
$this->file->next();
return $this->file->current();
}
}

View File

@@ -0,0 +1,67 @@
<?php
namespace Spatie\Backtrace\CodeSnippets;
class LaravelSerializableClosureSnippetProvider implements SnippetProvider
{
/** @var array<string> */
protected $lines;
/** @var int */
protected $counter = 0;
public function __construct(string $snippet)
{
$this->lines = preg_split("/\r\n|\n|\r/", $snippet);
$this->cleanupLines();
}
public function numberOfLines(): int
{
return count($this->lines);
}
public function getLine(?int $lineNumber = null): string
{
if (is_null($lineNumber)) {
return $this->getNextLine();
}
$this->counter = $lineNumber - 1;
return $this->lines[$lineNumber - 1];
}
public function getNextLine(): string
{
$this->counter++;
if ($this->counter >= count($this->lines)) {
return '';
}
return $this->lines[$this->counter];
}
protected function cleanupLines(): void
{
$spacesOrTabsToRemove = PHP_INT_MAX;
for ($i = 1; $i < count($this->lines); $i++) {
if (empty($this->lines[$i])) {
continue;
}
$spacesOrTabsToRemove = min(strspn($this->lines[$i], " \t"), $spacesOrTabsToRemove);
}
if ($spacesOrTabsToRemove === PHP_INT_MAX) {
$spacesOrTabsToRemove = 0;
}
for ($i = 1; $i < count($this->lines); $i++) {
$this->lines[$i] = substr($this->lines[$i], $spacesOrTabsToRemove);
}
}
}

View File

@@ -0,0 +1,21 @@
<?php
namespace Spatie\Backtrace\CodeSnippets;
class NullSnippetProvider implements SnippetProvider
{
public function numberOfLines(): int
{
return 1;
}
public function getLine(?int $lineNumber = null): string
{
return $this->getNextLine();
}
public function getNextLine(): string
{
return "File not found for code snippet";
}
}

View File

@@ -0,0 +1,12 @@
<?php
namespace Spatie\Backtrace\CodeSnippets;
interface SnippetProvider
{
public function numberOfLines(): int;
public function getLine(?int $lineNumber = null): string;
public function getNextLine(): string;
}

110
vendor/spatie/backtrace/src/Frame.php vendored Executable file
View File

@@ -0,0 +1,110 @@
<?php
namespace Spatie\Backtrace;
use Spatie\Backtrace\CodeSnippets\CodeSnippet;
use Spatie\Backtrace\CodeSnippets\FileSnippetProvider;
use Spatie\Backtrace\CodeSnippets\LaravelSerializableClosureSnippetProvider;
use Spatie\Backtrace\CodeSnippets\NullSnippetProvider;
use Spatie\Backtrace\CodeSnippets\SnippetProvider;
class Frame
{
/** @var string */
public $file;
/** @var string|null */
public $trimmedFilePath;
/** @var int */
public $lineNumber;
/** @var array|null */
public $arguments = null;
/** @var bool */
public $applicationFrame;
/** @var string|null */
public $method;
/** @var string|null */
public $class;
/** @var object|null */
public $object;
/** @var string|null */
protected $textSnippet;
public function __construct(
string $file,
int $lineNumber,
?array $arguments,
?string $method = null,
?string $class = null,
?object $object = null,
bool $isApplicationFrame = false,
?string $textSnippet = null,
?string $trimmedFilePath = null
) {
$this->file = $file;
$this->trimmedFilePath = $trimmedFilePath;
$this->lineNumber = $lineNumber;
$this->arguments = $arguments;
$this->method = $method;
$this->class = $class;
$this->object = $object;
$this->applicationFrame = $isApplicationFrame;
$this->textSnippet = $textSnippet;
}
public function getSnippet(int $lineCount): array
{
return (new CodeSnippet())
->surroundingLine($this->lineNumber)
->snippetLineCount($lineCount)
->get($this->getCodeSnippetProvider());
}
public function getSnippetAsString(int $lineCount): string
{
return (new CodeSnippet())
->surroundingLine($this->lineNumber)
->snippetLineCount($lineCount)
->getAsString($this->getCodeSnippetProvider());
}
public function getSnippetProperties(int $lineCount): array
{
$snippet = $this->getSnippet($lineCount);
return array_map(function (int $lineNumber) use ($snippet) {
return [
'line_number' => $lineNumber,
'text' => $snippet[$lineNumber],
];
}, array_keys($snippet));
}
protected function getCodeSnippetProvider(): SnippetProvider
{
if ($this->textSnippet) {
return new LaravelSerializableClosureSnippetProvider($this->textSnippet);
}
if (@file_exists($this->file)) {
return new FileSnippetProvider($this->file);
}
return new NullSnippetProvider();
}
}

21
vendor/spatie/browsershot/LICENSE.md vendored Executable file
View File

@@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) Spatie bvba <info@spatie.be>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

118
vendor/spatie/browsershot/README.md vendored Executable file
View File

@@ -0,0 +1,118 @@
<div align="left">
<a href="https://spatie.be/open-source?utm_source=github&utm_medium=banner&utm_campaign=browsershot">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://spatie.be/packages/header/browsershot/html/dark.webp">
<img alt="Logo for Browsershot" src="https://spatie.be/packages/header/browsershot/html/light.webp" height="190">
</picture>
</a>
<h1>Render web pages to an image or PDF with Puppeteer</h1>
[![Latest Version](https://img.shields.io/github/release/spatie/browsershot.svg?style=flat-square)](https://github.com/spatie/browsershot/releases)
[![MIT Licensed](https://img.shields.io/badge/license-MIT-brightgreen.svg?style=flat-square)](LICENSE.md)
[![run-tests](https://img.shields.io/github/actions/workflow/status/spatie/browsershot/run-tests.yml?label=tests&style=flat-square)](https://github.com/spatie/browsershot/actions)
[![Total Downloads](https://img.shields.io/packagist/dt/spatie/browsershot.svg?style=flat-square)](https://packagist.org/packages/spatie/browsershot)
</div>
The package can convert a web page to an image or PDF. The conversion is done behind the scenes by [Puppeteer](https://github.com/GoogleChrome/puppeteer) which runs a headless version of Google Chrome.
Here's a quick example:
```php
use Spatie\Browsershot\Browsershot;
// an image will be saved
Browsershot::url('https://example.com')->save($pathToImage);
```
It will save a PDF if the path passed to the `save` method has a `pdf` extension.
```php
// a pdf will be saved
Browsershot::url('https://example.com')->save('example.pdf');
```
You can also use an arbitrary html input, simply replace the `url` method with `html`:
```php
Browsershot::html('<h1>Hello world!!</h1>')->save('example.pdf');
```
If your HTML input is already in a file locally use the :
```php
Browsershot::htmlFromFilePath('/local/path/to/file.html')->save('example.pdf');
```
Browsershot also can get the body of an html page after JavaScript has been executed:
```php
Browsershot::url('https://example.com')->bodyHtml(); // returns the html of the body
```
If you wish to retrieve an array list with all of the requests that the page triggered you can do so:
```php
$requests = Browsershot::url('https://example.com')
->triggeredRequests();
foreach ($requests as $request) {
$url = $request['url']; //https://example.com/
}
```
To use Chrome's new [headless mode](https://developers.google.com/web/updates/2017/04/headless-chrome) pass the `newHeadless` method:
```php
Browsershot::url('https://example.com')->newHeadless()->save($pathToImage);
```
## Support us
Learn how to create a package like this one, by watching our premium video course:
[![Laravel Package training](https://spatie.be/github/package-training.jpg)](https://laravelpackage.training)
We invest a lot of resources into creating [best in class open source packages](https://spatie.be/open-source). You can support us by [buying one of our paid products](https://spatie.be/open-source/support-us).
We highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using. You'll find our address on [our contact page](https://spatie.be/about-us). We publish all received postcards on [our virtual postcard wall](https://spatie.be/open-source/postcards).
## Documentation
All documentation is available [on our documentation site](https://spatie.be/docs/browsershot).
## Testing
For running the testsuite, you'll need to have Puppeteer installed. Pleaser refer to the Browsershot requirements [here](https://spatie.be/docs/browsershot/v4/requirements). Usually `npm -g i puppeteer` will do the trick.
Additionally, you'll need the `pdftotext` CLI which is part of the poppler-utils package. More info can be found in in the [spatie/pdf-to-text readme](https://github.com/spatie/pdf-to-text?tab=readme-ov-file#requirements). Usually `brew install poppler-utils` will suffice.
Finally run the tests with:
```bash
composer test
```
## Contributing
Please see [CONTRIBUTING](https://github.com/spatie/.github/blob/main/CONTRIBUTING.md) for details.
## Security
If you've found a bug regarding security please mail [security@spatie.be](mailto:security@spatie.be) instead of using the issue tracker.
## Alternatives
If you're not able to install Node and Puppeteer, take a look at [v2 of browsershot](https://github.com/spatie/browsershot/tree/2.4.1), which uses Chrome headless CLI to take a screenshot. `v2` is not maintained anymore, but should work pretty well.
If using headless Chrome does not work for you take a look at at `v1` of this package which uses the abandoned `PhantomJS` binary.
## Credits
- [Freek Van der Herten](https://github.com/freekmurze)
- [All Contributors](../../contributors)
## License
The MIT License (MIT). Please see [License File](LICENSE.md) for more information.

445
vendor/spatie/browsershot/bin/browser.cjs vendored Executable file
View File

@@ -0,0 +1,445 @@
const fs = require('fs');
const URL = require('url').URL;
const URLParse = require('url').parse;
const [, , ...args] = process.argv;
/**
* There are two ways for Browsershot to communicate with puppeteer:
* - By giving a options JSON dump as an argument
* - Or by providing a temporary file with the options JSON dump,
* the path to this file is then given as an argument with the flag -f
*/
const request = args[0].startsWith('-f ')
? JSON.parse(fs.readFileSync(new URL(args[0].substring(3))))
: JSON.parse(args[0]);
const requestsList = [];
const redirectHistory = [];
const consoleMessages = [];
const failedRequests = [];
const pageErrors = [];
const getOutput = async (request, page = null) => {
let output = {
requestsList,
consoleMessages,
failedRequests,
redirectHistory,
pageErrors,
};
if (
![
'requestsList',
'consoleMessages',
'failedRequests',
'redirectHistory',
'pageErrors',
].includes(request.action) &&
page
) {
if (request.action == 'evaluate') {
output.result = await page.evaluate(request.options.pageFunction);
} else {
const result = await page[request.action](request.options);
// Ignore output result when saving to a file
output.result = request.options.path
? ''
: (result instanceof Uint8Array ? Buffer.from(result) : result).toString('base64');
}
}
if (page) {
return JSON.stringify(output);
}
// this will allow adding additional error info (only reach this point when there's an exception)
return output;
};
const callChrome = async pup => {
let browser;
let page;
let remoteInstance;
const puppet = (pup || require('puppeteer'));
try {
if (request.options.remoteInstanceUrl || request.options.browserWSEndpoint ) {
// default options
let options = {
acceptInsecureCerts: request.options.acceptInsecureCerts
};
// choose only one method to connect to the browser instance
if ( request.options.remoteInstanceUrl ) {
options.browserURL = request.options.remoteInstanceUrl;
} else if ( request.options.browserWSEndpoint ) {
options.browserWSEndpoint = request.options.browserWSEndpoint;
}
try {
browser = await puppet.connect( options );
remoteInstance = true;
} catch (exception) { /** does nothing. fallbacks to launching a chromium instance */}
}
if (!browser) {
browser = await puppet.launch({
headless: request.options.newHeadless ? true : 'shell',
acceptInsecureCerts: request.options.acceptInsecureCerts,
executablePath: request.options.executablePath,
args: request.options.args || [],
pipe: request.options.pipe || false,
env: {
...(request.options.env || {}),
...process.env
},
protocolTimeout: request.options.protocolTimeout ?? 30000,
});
}
page = await browser.newPage();
if (request.options && request.options.disableJavascript) {
await page.setJavaScriptEnabled(false);
}
await page.setRequestInterception(true);
const contentUrl = request.options.contentUrl;
const parsedContentUrl = contentUrl ? contentUrl.replace(/\/$/, "") : undefined;
let pageContent;
if (contentUrl) {
pageContent = fs.readFileSync(request.url.replace('file://', ''));
request.url = contentUrl;
}
page.on('console', (message) =>
consoleMessages.push({
type: message.type(),
message: message.text(),
location: message.location(),
stackTrace: message.stackTrace(),
})
);
page.on('pageerror', (msg) => {
pageErrors.push({
name: msg?.name || 'unknown error',
message: msg?.message || msg?.toString() || 'null'
});
});
page.on('response', function (response) {
const frame = response.request().frame();
if (response.request().isNavigationRequest() && frame && frame.parentFrame() === null) {
redirectHistory.push({
url: response.request().url(),
status: response.status(),
reason: response.statusText(),
headers: response.headers()
})
}
if (response.status() >= 200 && response.status() <= 399) {
return;
}
failedRequests.push({
status: response.status(),
url: response.url(),
});
})
page.on('request', interceptedRequest => {
var headers = interceptedRequest.headers();
if (!request.options || !request.options.disableCaptureURLS) {
requestsList.push({
url: interceptedRequest.url(),
});
}
if (request.options && request.options.disableImages) {
if (interceptedRequest.resourceType() === 'image') {
interceptedRequest.abort();
return;
}
}
if (request.options && request.options.blockDomains) {
const hostname = URLParse(interceptedRequest.url()).hostname;
if (request.options.blockDomains.includes(hostname)) {
interceptedRequest.abort();
return;
}
}
if (request.options && request.options.blockUrls) {
for (const element of request.options.blockUrls) {
if (interceptedRequest.url().indexOf(element) >= 0) {
interceptedRequest.abort();
return;
}
}
}
if (request.options && request.options.disableRedirects) {
if (interceptedRequest.isNavigationRequest() && interceptedRequest.redirectChain().length) {
interceptedRequest.abort();
return
}
}
if (request.options && request.options.extraNavigationHTTPHeaders) {
// Do nothing in case of non-navigation requests.
if (interceptedRequest.isNavigationRequest()) {
headers = Object.assign({}, headers, request.options.extraNavigationHTTPHeaders);
}
}
if (pageContent) {
const interceptedUrl = interceptedRequest.url().replace(/\/$/, "");
// if content url matches the intercepted request url, will return the content fetched from the local file system
if (interceptedUrl === parsedContentUrl) {
interceptedRequest.respond({
headers,
body: pageContent,
});
return;
}
}
if (request.postParams) {
const postParamsArray = request.postParams;
const queryString = Object.keys(postParamsArray)
.map(key => `${key}=${postParamsArray[key]}`)
.join('&');
interceptedRequest.continue({
method: "POST",
postData: queryString,
headers: {
...interceptedRequest.headers(),
"Content-Type": "application/x-www-form-urlencoded"
}
});
return;
}
interceptedRequest.continue({ headers });
});
if (request.options && request.options.dismissDialogs) {
page.on('dialog', async dialog => {
await dialog.dismiss();
});
}
if (request.options && request.options.userAgent) {
await page.setUserAgent(request.options.userAgent);
}
if (request.options && request.options.device) {
const devices = puppet.KnownDevices;
const device = devices[request.options.device];
await page.emulate(device);
}
if (request.options && request.options.emulateMedia) {
await page.emulateMediaType(request.options.emulateMedia);
}
if (request.options && request.options.emulateMediaFeatures) {
await page.emulateMediaFeatures(JSON.parse(request.options.emulateMediaFeatures));
}
if (request.options && request.options.viewport) {
await page.setViewport(request.options.viewport);
}
if (request.options && request.options.extraHTTPHeaders) {
await page.setExtraHTTPHeaders(request.options.extraHTTPHeaders);
}
if (request.options && request.options.authentication) {
await page.authenticate(request.options.authentication);
}
if (request.options && request.options.cookies) {
await page.setCookie(...request.options.cookies);
}
if (request.options && request.options.timeout) {
await page.setDefaultNavigationTimeout(request.options.timeout);
}
const requestOptions = {};
if (request.options && request.options.networkIdleTimeout) {
requestOptions.waitUntil = 'networkidle';
requestOptions.networkIdleTimeout = request.options.networkIdleTimeout;
} else if (request.options && request.options.waitUntil) {
requestOptions.waitUntil = request.options.waitUntil;
}
const response = await page.goto(request.url, requestOptions);
if (request.options.preventUnsuccessfulResponse) {
const status = response.status()
if (status >= 400 && status < 600) {
throw {type: "UnsuccessfulResponse", status};
}
}
if (request.options && request.options.disableImages) {
await page.evaluate(() => {
let images = document.getElementsByTagName('img');
while (images.length > 0) {
images[0].parentNode.removeChild(images[0]);
}
});
}
if (request.options && request.options.types) {
for (let i = 0, len = request.options.types.length; i < len; i++) {
let typeOptions = request.options.types[i];
await page.type(typeOptions.selector, typeOptions.text, {
'delay': typeOptions.delay,
});
}
}
if (request.options && request.options.selects) {
for (let i = 0, len = request.options.selects.length; i < len; i++) {
let selectOptions = request.options.selects[i];
await page.select(selectOptions.selector, selectOptions.value);
}
}
if (request.options && request.options.clicks) {
for (let i = 0, len = request.options.clicks.length; i < len; i++) {
let clickOptions = request.options.clicks[i];
await page.click(clickOptions.selector, {
'button': clickOptions.button,
'clickCount': clickOptions.clickCount,
'delay': clickOptions.delay,
});
}
}
if (request.options && request.options.addStyleTag) {
await page.addStyleTag(JSON.parse(request.options.addStyleTag));
}
if (request.options && request.options.addScriptTag) {
await page.addScriptTag(JSON.parse(request.options.addScriptTag));
}
if (request.options.delay) {
await new Promise(r => setTimeout(r, request.options.delay));
}
if (request.options.initialPageNumber) {
await page.evaluate((initialPageNumber) => {
window.pageStart = initialPageNumber;
const style = document.createElement('style');
style.type = 'text/css';
style.innerHTML = '.empty-page { page-break-after: always; visibility: hidden; }';
document.getElementsByTagName('head')[0].appendChild(style);
const emptyPages = Array.from({length: window.pageStart}).map(() => {
const emptyPage = document.createElement('div');
emptyPage.className = "empty-page";
emptyPage.textContent = "empty";
return emptyPage;
});
document.body.prepend(...emptyPages);
}, request.options.initialPageNumber);
}
if (request.options.function) {
let functionOptions = {
polling: request.options.functionPolling,
timeout: request.options.functionTimeout || request.options.timeout
};
await page.waitForFunction(request.options.function, functionOptions);
}
if (request.options.waitForSelector) {
await page.waitForSelector(request.options.waitForSelector, (request.options.waitForSelectorOptions ? request.options.waitForSelectorOptions : undefined));
}
if (request.options.selector) {
var element;
const index = request.options.selectorIndex || 0;
if(index){
element = await page.$$(request.options.selector);
if(!element.length || typeof element[index] === 'undefined'){
element = null;
}else{
element = element[index];
}
}else{
element = await page.$(request.options.selector);
}
if (element === null) {
throw {type: 'ElementNotFound'};
}
request.options.clip = await element.boundingBox();
}
console.log(await getOutput(request, page));
if (remoteInstance && page) {
await page.close();
}
await (remoteInstance ? browser.disconnect() : browser.close());
} catch (exception) {
if (browser) {
if (remoteInstance && page) {
await page.close();
}
await (remoteInstance ? browser.disconnect() : browser.close());
}
const output = await getOutput(request);
if (exception.type === 'UnsuccessfulResponse') {
output.exception = exception.toString();
console.error(exception.status);
console.log(JSON.stringify(output));
process.exit(3);
}
output.exception = exception.toString();
console.error(exception);
console.log(JSON.stringify(output));
if (exception.type === 'ElementNotFound') {
process.exit(2);
}
process.exit(1);
}
};
if (require.main === module) {
callChrome();
}
exports.callChrome = callChrome;

56
vendor/spatie/browsershot/composer.json vendored Executable file
View File

@@ -0,0 +1,56 @@
{
"name": "spatie/browsershot",
"description": "Convert a webpage to an image or pdf using headless Chrome",
"homepage": "https://github.com/spatie/browsershot",
"keywords": [
"convert",
"webpage",
"image",
"pdf",
"screenshot",
"chrome",
"headless",
"puppeteer"
],
"license": "MIT",
"authors": [
{
"name": "Freek Van der Herten",
"email": "freek@spatie.be",
"homepage": "https://github.com/freekmurze",
"role": "Developer"
}
],
"require": {
"php": "^8.2",
"spatie/temporary-directory": "^2.0",
"symfony/process": "^6.0|^7.0",
"ext-json": "*",
"ext-fileinfo": "*"
},
"require-dev": {
"pestphp/pest": "^3.0",
"spatie/image": "^3.6",
"spatie/pdf-to-text": "^1.52",
"spatie/phpunit-snapshot-assertions": "^4.2.3|^5.0"
},
"autoload": {
"psr-4": {
"Spatie\\Browsershot\\": "src"
}
},
"autoload-dev": {
"psr-4": {
"Spatie\\Browsershot\\Test\\": "tests"
}
},
"scripts": {
"test": "vendor/bin/pest"
},
"config": {
"sort-packages": true,
"allow-plugins": {
"pestphp/pest-plugin": true
}
}
}

1233
vendor/spatie/browsershot/src/Browsershot.php vendored Executable file

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,131 @@
<?php
namespace Spatie\Browsershot;
/**
* Class with all outputs generated by puppeteer
*
* All present data is always relative to the last browser call.
*
* This object contains:
*
* - consoleMessages: messages generated with console calls
* - requestsList: list of all requests made
* - failedRequests: list of all failed requests
* - result: result of the last operation called
* - exception: string representation of the exception generated, if any
* - pageErrors: list of all page errors generated during the current command
* - redirectHistory: list of all redirect in the page
*/
class ChromiumResult
{
protected string $result;
protected ?string $exception;
/** @var null|array{
* type: string,
* message: string,
* location: array,
* stackTrace: string
* }
*/
protected ?array $consoleMessages;
/** @var null|array{url: string} */
protected ?array $requestsList;
/** * @var null|array{
* status: int,
* url: string
* }
*/
protected ?array $failedRequests;
/** * @var null|array{
* name: string,
* message: string
* }
*/
protected ?array $pageErrors;
/** @var null|array{
* url: string,
* status: int,
* statusText: string,
* headers: array
* }
*/
protected ?array $redirectHistory;
public function __construct(?array $output)
{
$this->result = $output['result'] ?? '';
$this->exception = $output['exception'] ?? null;
$this->consoleMessages = $output['consoleMessages'] ?? null;
$this->requestsList = $output['requestsList'] ?? null;
$this->failedRequests = $output['failedRequests'] ?? null;
$this->pageErrors = $output['pageErrors'] ?? null;
$this->redirectHistory = $output['redirectHistory'] ?? null;
}
public function getResult(): string
{
return $this->result;
}
public function getException(): ?string
{
return $this->exception;
}
/** @return null|array{
* type: string,
* message: string,
* location: array,
* stackTrace: string
* }
*/
public function getConsoleMessages(): ?array
{
return $this->consoleMessages;
}
/**
* @return null|array{url: string}
*/
public function getRequestsList(): ?array
{
return $this->requestsList;
}
/**
* @return null|array{status: int, url: string}
*/
public function getFailedRequests(): ?array
{
return $this->failedRequests;
}
/** @return null|array{
* name: string,
* message: string
* }
*/
public function getPageErrors(): ?array
{
return $this->pageErrors;
}
/** @return null|array{
* url: string,
* status: int,
* statusText: string,
* headers: array
* }
*/
public function getRedirectHistory(): ?array
{
return $this->redirectHistory;
}
}

View File

@@ -0,0 +1,9 @@
<?php
namespace Spatie\Browsershot\Enums;
enum Polling: string
{
case RequestAnimationFrame = 'raf';
case Mutation = 'mutation';
}

View File

@@ -0,0 +1,30 @@
<?php
namespace Spatie\Browsershot\Exceptions;
use Exception;
class CouldNotTakeBrowsershot extends Exception
{
public static function chromeOutputEmpty(string $screenShotPath, string $output, array $command = []): static
{
$command = json_encode($command);
$message = <<<CONSOLE
For some reason Chrome did not write a file at `{$screenShotPath}`.
Command
=======
{$command}
Output
======
{$output}
CONSOLE;
return new static($message);
}
public static function outputFileDidNotHaveAnExtension(string $path): static
{
return new static("The given path `{$path}` did not contain an extension. Please append an extension.");
}
}

View File

@@ -0,0 +1,13 @@
<?php
namespace Spatie\Browsershot\Exceptions;
use Exception;
class ElementNotFound extends Exception
{
public static function make(string $selector): static
{
return new static("The given selector `{$selector} did not match any elements");
}
}

View File

@@ -0,0 +1,13 @@
<?php
namespace Spatie\Browsershot\Exceptions;
use Exception;
class FileDoesNotExistException extends Exception
{
public static function make(string $file): static
{
return new static("The file `{$file}` does not exist");
}
}

View File

@@ -0,0 +1,18 @@
<?php
namespace Spatie\Browsershot\Exceptions;
use Exception;
class FileUrlNotAllowed extends Exception
{
public static function make(): static
{
return new static('An URL is not allow to start with file:// or file:/');
}
public static function urlCannotBeParsed(string $url): static
{
return new static("The given URL `{$url}` is not a valid URL");
}
}

View File

@@ -0,0 +1,13 @@
<?php
namespace Spatie\Browsershot\Exceptions;
use Exception;
class HtmlIsNotAllowedToContainFile extends Exception
{
public static function make(): static
{
return new static('The specified HTML contains `file://` or `file:/`. This is not allowed.');
}
}

View File

@@ -0,0 +1,13 @@
<?php
namespace Spatie\Browsershot\Exceptions;
use Exception;
class UnsuccessfulResponse extends Exception
{
public static function make(string $url, string|int $code): static
{
return new static("The given url `{$url}` responds with code {$code}");
}
}

View File

@@ -0,0 +1,57 @@
<?php
namespace Spatie\Browsershot;
use Composer\InstalledVersions;
use Exception;
use Spatie\Image\Image;
class ImageManipulations
{
protected array $manipulations = [];
public function __call(string $method, array $parameters): self
{
$this->addManipulation($method, $parameters);
return $this;
}
public function addManipulation(string $name, array $parameters = []): self
{
$this->manipulations[$name] = $parameters;
return $this;
}
public function apply(string $path): void
{
$this->ensureImageDependencyIsInstalled();
$image = Image::load($path);
foreach ($this->manipulations as $manipulationName => $parameters) {
$image->$manipulationName(...$parameters);
}
$image->save($path);
}
public function isEmpty(): bool
{
return count($this->manipulations) === 0;
}
public function ensureImageDependencyIsInstalled(): void
{
if (! InstalledVersions::isInstalled('spatie/image')) {
throw new Exception('The spatie/image package is required to perform image manipulations. Please install it by running `composer require spatie/image`');
}
$installedVersion = InstalledVersions::getVersion('spatie/image');
if (version_compare($installedVersion, '3.0.0', '<')) {
throw new Exception("The spatie/image package must be at least version 3.0.0 to perform image manipulations. Your current version is `{$installedVersion}`");
}
}
}

21
vendor/spatie/crawler/LICENSE.md vendored Executable file
View File

@@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) Spatie bvba <info@spatie.be>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

482
vendor/spatie/crawler/README.md vendored Executable file
View File

@@ -0,0 +1,482 @@
<div align="left">
<a href="https://spatie.be/open-source?utm_source=github&utm_medium=banner&utm_campaign=crawler">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://spatie.be/packages/header/crawler/html/dark.webp">
<img alt="Logo for crawler" src=" https://spatie.be/packages/header/crawler/html/light.webp" height="190">
</picture>
</a>
<h1>🕸 Crawl the web using PHP 🕷</h1>
[![Latest Version on Packagist](https://img.shields.io/packagist/v/spatie/crawler.svg?style=flat-square)](https://packagist.org/packages/spatie/crawler)
[![MIT Licensed](https://img.shields.io/badge/license-MIT-brightgreen.svg?style=flat-square)](LICENSE.md)
![Tests](https://github.com/spatie/crawler/workflows/Tests/badge.svg)
[![Total Downloads](https://img.shields.io/packagist/dt/spatie/crawler.svg?style=flat-square)](https://packagist.org/packages/spatie/crawler)
</div>
This package provides a class to crawl links on a website. Under the hood Guzzle promises are used to [crawl multiple urls concurrently](http://docs.guzzlephp.org/en/latest/quickstart.html?highlight=pool#concurrent-requests).
Because the crawler can execute JavaScript, it can crawl JavaScript rendered sites. Under the hood [Chrome and Puppeteer](https://github.com/spatie/browsershot) are used to power this feature.
## Support us
[<img src="https://github-ads.s3.eu-central-1.amazonaws.com/crawler.jpg?t=1" width="419px" />](https://spatie.be/github-ad-click/crawler)
We invest a lot of resources into creating [best in class open source packages](https://spatie.be/open-source). You can support us by [buying one of our paid products](https://spatie.be/open-source/support-us).
We highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using. You'll find our address on [our contact page](https://spatie.be/about-us). We publish all received postcards on [our virtual postcard wall](https://spatie.be/open-source/postcards).
## Installation
This package can be installed via Composer:
``` bash
composer require spatie/crawler
```
## Usage
The crawler can be instantiated like this
```php
use Spatie\Crawler\Crawler;
Crawler::create()
->setCrawlObserver(<class that extends \Spatie\Crawler\CrawlObservers\CrawlObserver>)
->startCrawling($url);
```
The argument passed to `setCrawlObserver` must be an object that extends the `\Spatie\Crawler\CrawlObservers\CrawlObserver` abstract class:
```php
namespace Spatie\Crawler\CrawlObservers;
use GuzzleHttp\Exception\RequestException;
use Psr\Http\Message\ResponseInterface;
use Psr\Http\Message\UriInterface;
abstract class CrawlObserver
{
/*
* Called when the crawler will crawl the url.
*/
public function willCrawl(UriInterface $url, ?string $linkText): void
{
}
/*
* Called when the crawler has crawled the given url successfully.
*/
abstract public function crawled(
UriInterface $url,
ResponseInterface $response,
?UriInterface $foundOnUrl = null,
?string $linkText,
): void;
/*
* Called when the crawler had a problem crawling the given url.
*/
abstract public function crawlFailed(
UriInterface $url,
RequestException $requestException,
?UriInterface $foundOnUrl = null,
?string $linkText = null,
): void;
/**
* Called when the crawl has ended.
*/
public function finishedCrawling(): void
{
}
}
```
### Using multiple observers
You can set multiple observers with `setCrawlObservers`:
```php
Crawler::create()
->setCrawlObservers([
<class that extends \Spatie\Crawler\CrawlObservers\CrawlObserver>,
<class that extends \Spatie\Crawler\CrawlObservers\CrawlObserver>,
...
])
->startCrawling($url);
```
Alternatively you can set multiple observers one by one with `addCrawlObserver`:
```php
Crawler::create()
->addCrawlObserver(<class that extends \Spatie\Crawler\CrawlObservers\CrawlObserver>)
->addCrawlObserver(<class that extends \Spatie\Crawler\CrawlObservers\CrawlObserver>)
->addCrawlObserver(<class that extends \Spatie\Crawler\CrawlObservers\CrawlObserver>)
->startCrawling($url);
```
### Executing JavaScript
By default, the crawler will not execute JavaScript. This is how you can enable the execution of JavaScript:
```php
Crawler::create()
->executeJavaScript()
...
```
In order to make it possible to get the body html after the javascript has been executed, this package depends on
our [Browsershot](https://github.com/spatie/browsershot) package.
This package uses [Puppeteer](https://github.com/puppeteer/puppeteer) under the hood. Here are some pointers on [how to install it on your system](https://spatie.be/docs/browsershot/v2/requirements).
Browsershot will make an educated guess as to where its dependencies are installed on your system.
By default, the Crawler will instantiate a new Browsershot instance. You may find the need to set a custom created instance using the `setBrowsershot(Browsershot $browsershot)` method.
```php
Crawler::create()
->setBrowsershot($browsershot)
->executeJavaScript()
...
```
Note that the crawler will still work even if you don't have the system dependencies required by Browsershot.
These system dependencies are only required if you're calling `executeJavaScript()`.
### Filtering certain urls
You can tell the crawler not to visit certain urls by using the `setCrawlProfile`-function. That function expects
an object that extends `Spatie\Crawler\CrawlProfiles\CrawlProfile`:
```php
/*
* Determine if the given url should be crawled.
*/
public function shouldCrawl(UriInterface $url): bool;
```
This package comes with three `CrawlProfiles` out of the box:
- `CrawlAllUrls`: this profile will crawl all urls on all pages including urls to an external site.
- `CrawlInternalUrls`: this profile will only crawl the internal urls on the pages of a host.
- `CrawlSubdomains`: this profile will only crawl the internal urls and its subdomains on the pages of a host.
### Custom link extraction
You can customize how links are extracted from a page by passing a custom `UrlParser` to the crawler.
```php
Crawler::create()
->setUrlParserClass(<class that implements \Spatie\Crawler\UrlParsers\UrlParser>::class)
...
```
By default, the `LinkUrlParser` is used. This parser will extract all links from the `href` attribute of `a` tags.
There is also a built-in `SitemapUrlParser` that will extract & crawl all links from a sitemap. It does support sitemap index files.
```php
Crawler::create()
->setUrlParserClass(SitemapUrlParser::class)
...
```
### Ignoring robots.txt and robots meta
By default, the crawler will respect robots data. It is possible to disable these checks like so:
```php
Crawler::create()
->ignoreRobots()
...
```
Robots data can come from either a `robots.txt` file, meta tags or response headers.
More information on the spec can be found here: [http://www.robotstxt.org/](http://www.robotstxt.org/).
Parsing robots data is done by our package [spatie/robots-txt](https://github.com/spatie/robots-txt).
### Accept links with rel="nofollow" attribute
By default, the crawler will reject all links containing attribute rel="nofollow". It is possible to disable these checks like so:
```php
Crawler::create()
->acceptNofollowLinks()
...
```
### Using a custom User Agent ###
In order to respect robots.txt rules for a custom User Agent you can specify your own custom User Agent.
```php
Crawler::create()
->setUserAgent('my-agent')
```
You can add your specific crawl rule group for 'my-agent' in robots.txt. This example disallows crawling the entire site for crawlers identified by 'my-agent'.
```txt
// Disallow crawling for my-agent
User-agent: my-agent
Disallow: /
```
## Setting the number of concurrent requests
To improve the speed of the crawl the package concurrently crawls 10 urls by default. If you want to change that number you can use the `setConcurrency` method.
```php
Crawler::create()
->setConcurrency(1) // now all urls will be crawled one by one
```
## Defining Crawl and Time Limits
By default, the crawler continues until it has crawled every page it can find. This behavior might cause issues if you are working in an environment with limitations such as a serverless environment.
The crawl behavior can be controlled with the following two options:
- **Total Crawl Limit** (`setTotalCrawlLimit`): This limit defines the maximal count of URLs to crawl.
- **Current Crawl Limit** (`setCurrentCrawlLimit`): This defines how many URLs are processed during the current crawl.
- **Total Execution Time Limit** (`setTotalExecutionTimeLimit`): This limit defines the maximal execution time of the crawl.
- **Current Execution Time Limit** (`setCurrentExecutionTimeLimit`): This limits the execution time of the current crawl.
Let's take a look at some examples to clarify the difference between `setTotalCrawlLimit` and `setCurrentCrawlLimit`.
The difference between `setTotalExecutionTimeLimit` and `setCurrentExecutionTimeLimit` will be the same.
### Example 1: Using the total crawl limit
The `setTotalCrawlLimit` method allows you to limit the total number of URLs to crawl, no matter how often you call the crawler.
```php
$queue = <your selection/implementation of a queue>;
// Crawls 5 URLs and ends.
Crawler::create()
->setCrawlQueue($queue)
->setTotalCrawlLimit(5)
->startCrawling($url);
// Doesn't crawl further as the total limit is reached.
Crawler::create()
->setCrawlQueue($queue)
->setTotalCrawlLimit(5)
->startCrawling($url);
```
### Example 2: Using the current crawl limit
The `setCurrentCrawlLimit` will set a limit on how many URls will be crawled per execution. This piece of code will process 5 pages with each execution, without a total limit of pages to crawl.
```php
$queue = <your selection/implementation of a queue>;
// Crawls 5 URLs and ends.
Crawler::create()
->setCrawlQueue($queue)
->setCurrentCrawlLimit(5)
->startCrawling($url);
// Crawls the next 5 URLs and ends.
Crawler::create()
->setCrawlQueue($queue)
->setCurrentCrawlLimit(5)
->startCrawling($url);
```
### Example 3: Combining the total and crawl limit
Both limits can be combined to control the crawler:
```php
$queue = <your selection/implementation of a queue>;
// Crawls 5 URLs and ends.
Crawler::create()
->setCrawlQueue($queue)
->setTotalCrawlLimit(10)
->setCurrentCrawlLimit(5)
->startCrawling($url);
// Crawls the next 5 URLs and ends.
Crawler::create()
->setCrawlQueue($queue)
->setTotalCrawlLimit(10)
->setCurrentCrawlLimit(5)
->startCrawling($url);
// Doesn't crawl further as the total limit is reached.
Crawler::create()
->setCrawlQueue($queue)
->setTotalCrawlLimit(10)
->setCurrentCrawlLimit(5)
->startCrawling($url);
```
### Example 4: Crawling across requests
You can use the `setCurrentCrawlLimit` to break up long running crawls. The following example demonstrates a (simplified) approach. It's made up of an initial request and any number of follow-up requests continuing the crawl.
#### Initial Request
To start crawling across different requests, you will need to create a new queue of your selected queue-driver. Start by passing the queue-instance to the crawler. The crawler will start filling the queue as pages are processed and new URLs are discovered. Serialize and store the queue reference after the crawler has finished (using the current crawl limit).
```php
// Create a queue using your queue-driver.
$queue = <your selection/implementation of a queue>;
// Crawl the first set of URLs
Crawler::create()
->setCrawlQueue($queue)
->setCurrentCrawlLimit(10)
->startCrawling($url);
// Serialize and store your queue
$serializedQueue = serialize($queue);
```
#### Subsequent Requests
For any following requests you will need to unserialize your original queue and pass it to the crawler:
```php
// Unserialize queue
$queue = unserialize($serializedQueue);
// Crawls the next set of URLs
Crawler::create()
->setCrawlQueue($queue)
->setCurrentCrawlLimit(10)
->startCrawling($url);
// Serialize and store your queue
$serialized_queue = serialize($queue);
```
The behavior is based on the information in the queue. Only if the same queue-instance is passed in the behavior works as described. When a completely new queue is passed in, the limits of previous crawls -even for the same website- won't apply.
An example with more details can be found [here](https://github.com/spekulatius/spatie-crawler-cached-queue-example).
## Setting the maximum crawl depth
By default, the crawler continues until it has crawled every page of the supplied URL. If you want to limit the depth of the crawler you can use the `setMaximumDepth` method.
```php
Crawler::create()
->setMaximumDepth(2)
```
## Setting the maximum response size
Most html pages are quite small. But the crawler could accidentally pick up on large files such as PDFs and MP3s. To keep memory usage low in such cases the crawler will only use the responses that are smaller than 2 MB. If, when streaming a response, it becomes larger than 2 MB, the crawler will stop streaming the response. An empty response body will be assumed.
You can change the maximum response size.
```php
// let's use a 3 MB maximum.
Crawler::create()
->setMaximumResponseSize(1024 * 1024 * 3)
```
## Add a delay between requests
In some cases you might get rate-limited when crawling too aggressively. To circumvent this, you can use the `setDelayBetweenRequests()` method to add a pause between every request. This value is expressed in milliseconds.
```php
Crawler::create()
->setDelayBetweenRequests(150) // After every page crawled, the crawler will wait for 150ms
```
## Limiting which content-types to parse
By default, every found page will be downloaded (up to `setMaximumResponseSize()` in size) and parsed for additional links. You can limit which content-types should be downloaded and parsed by setting the `setParseableMimeTypes()` with an array of allowed types.
```php
Crawler::create()
->setParseableMimeTypes(['text/html', 'text/plain'])
```
This will prevent downloading the body of pages that have different mime types, like binary files, audio/video, ... that are unlikely to have links embedded in them. This feature mostly saves bandwidth.
## Using a custom crawl queue
When crawling a site the crawler will put urls to be crawled in a queue. By default, this queue is stored in memory using the built-in `ArrayCrawlQueue`.
When a site is very large you may want to store that queue elsewhere, maybe a database. In such cases, you can write your own crawl queue.
A valid crawl queue is any class that implements the `Spatie\Crawler\CrawlQueues\CrawlQueue`-interface. You can pass your custom crawl queue via the `setCrawlQueue` method on the crawler.
```php
Crawler::create()
->setCrawlQueue(<implementation of \Spatie\Crawler\CrawlQueues\CrawlQueue>)
```
Here
- [ArrayCrawlQueue](https://github.com/spatie/crawler/blob/master/src/CrawlQueues/ArrayCrawlQueue.php)
- [RedisCrawlQueue (third-party package)](https://github.com/repat/spatie-crawler-redis)
- [CacheCrawlQueue for Laravel (third-party package)](https://github.com/spekulatius/spatie-crawler-toolkit-for-laravel)
- [Laravel Model as Queue (third-party example app)](https://github.com/insign/spatie-crawler-queue-with-laravel-model)
## Change the default base url scheme
By default, the crawler will set the base url scheme to `http` if none. You have the ability to change that with `setDefaultScheme`.
```php
Crawler::create()
->setDefaultScheme('https')
```
## Changelog
Please see [CHANGELOG](CHANGELOG.md) for more information what has changed recently.
## Contributing
Please see [CONTRIBUTING](https://github.com/spatie/.github/blob/main/CONTRIBUTING.md) for details.
## Testing
First, install the Puppeteer dependency, or your tests will fail.
```
npm install puppeteer
```
To run the tests you'll have to start the included node based server first in a separate terminal window.
```bash
cd tests/server
npm install
node server.js
```
With the server running, you can start testing.
```bash
composer test
```
## Security
If you've found a bug regarding security please mail [security@spatie.be](mailto:security@spatie.be) instead of using the issue tracker.
## Postcardware
You're free to use this package, but if it makes it to your production environment we highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using.
Our address is: Spatie, Kruikstraat 22, 2018 Antwerp, Belgium.
We publish all received postcards [on our company website](https://spatie.be/en/opensource/postcards).
## Credits
- [Freek Van der Herten](https://github.com/freekmurze)
- [All Contributors](../../contributors)
## License
The MIT License (MIT). Please see [License File](LICENSE.md) for more information.

51
vendor/spatie/crawler/composer.json vendored Executable file
View File

@@ -0,0 +1,51 @@
{
"name": "spatie/crawler",
"description": "Crawl all internal links found on a website",
"keywords": [
"spatie",
"crawler",
"link",
"website"
],
"homepage": "https://github.com/spatie/crawler",
"license": "MIT",
"authors": [
{
"name": "Freek Van der Herten",
"email": "freek@spatie.be"
}
],
"require": {
"php": "^8.2",
"guzzlehttp/guzzle": "^7.3",
"guzzlehttp/psr7": "^2.0",
"illuminate/collections": "^10.0|^11.0|^12.0",
"nicmart/tree": "^0.9",
"spatie/browsershot": "^5.0.5",
"spatie/robots-txt": "^2.0",
"symfony/dom-crawler": "^6.0|^7.0"
},
"require-dev": {
"pestphp/pest": "^2.0|^3.0",
"spatie/ray": "^1.37"
},
"config": {
"sort-packages": true,
"allow-plugins": {
"pestphp/pest-plugin": true,
"phpstan/extension-installer": true
}
},
"autoload": {
"psr-4": {
"Spatie\\Crawler\\": "src"
}
},
"autoload-dev": {
"psr-4": {
"Spatie\\Crawler\\Test\\": "tests"
}
},
"minimum-stability": "dev",
"prefer-stable": true
}

17
vendor/spatie/crawler/phpunit.xml vendored Executable file
View File

@@ -0,0 +1,17 @@
<?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="vendor/phpunit/phpunit/phpunit.xsd"
bootstrap="vendor/autoload.php"
colors="true"
>
<testsuites>
<testsuite name="all">
<directory>tests</directory>
</testsuite>
</testsuites>
<source>
<include>
<directory>src</directory>
</include>
</source>
</phpunit>

View File

@@ -0,0 +1,40 @@
<?php
namespace Spatie\Crawler\CrawlObservers;
use GuzzleHttp\Exception\RequestException;
use Psr\Http\Message\ResponseInterface;
use Psr\Http\Message\UriInterface;
abstract class CrawlObserver
{
/*
* Called when the crawler will crawl the url.
*/
public function willCrawl(UriInterface $url, ?string $linkText): void {}
/*
* Called when the crawler has crawled the given url successfully.
*/
public function crawled(
UriInterface $url,
ResponseInterface $response,
?UriInterface $foundOnUrl = null,
?string $linkText = null,
): void {}
/*
* Called when the crawler had a problem crawling the given url.
*/
public function crawlFailed(
UriInterface $url,
RequestException $requestException,
?UriInterface $foundOnUrl = null,
?string $linkText = null,
): void {}
/*
* Called when the crawl has ended.
*/
public function finishedCrawling(): void {}
}

View File

@@ -0,0 +1,97 @@
<?php
namespace Spatie\Crawler\CrawlObservers;
use ArrayAccess;
use GuzzleHttp\Exception\RequestException;
use Iterator;
use Psr\Http\Message\ResponseInterface;
use Spatie\Crawler\CrawlUrl;
class CrawlObserverCollection implements ArrayAccess, Iterator
{
protected int $position;
public function __construct(protected array $observers = [])
{
$this->position = 0;
}
public function addObserver(CrawlObserver $observer): void
{
$this->observers[] = $observer;
}
public function crawled(CrawlUrl $crawlUrl, ResponseInterface $response): void
{
foreach ($this->observers as $crawlObserver) {
$crawlObserver->crawled(
$crawlUrl->url,
$response,
$crawlUrl->foundOnUrl,
$crawlUrl->linkText,
);
}
}
public function crawlFailed(CrawlUrl $crawlUrl, RequestException $exception): void
{
foreach ($this->observers as $crawlObserver) {
$crawlObserver->crawlFailed(
$crawlUrl->url,
$exception,
$crawlUrl->foundOnUrl,
$crawlUrl->linkText,
);
}
}
public function current(): mixed
{
return $this->observers[$this->position];
}
public function offsetGet(mixed $offset): mixed
{
return $this->observers[$offset] ?? null;
}
public function offsetSet(mixed $offset, mixed $value): void
{
if (is_null($offset)) {
$this->observers[] = $value;
} else {
$this->observers[$offset] = $value;
}
}
public function offsetExists(mixed $offset): bool
{
return isset($this->observers[$offset]);
}
public function offsetUnset(mixed $offset): void
{
unset($this->observers[$offset]);
}
public function next(): void
{
$this->position++;
}
public function key(): mixed
{
return $this->position;
}
public function valid(): bool
{
return isset($this->observers[$this->position]);
}
public function rewind(): void
{
$this->position = 0;
}
}

View File

@@ -0,0 +1,13 @@
<?php
namespace Spatie\Crawler\CrawlProfiles;
use Psr\Http\Message\UriInterface;
class CrawlAllUrls extends CrawlProfile
{
public function shouldCrawl(UriInterface $url): bool
{
return true;
}
}

View File

@@ -0,0 +1,25 @@
<?php
namespace Spatie\Crawler\CrawlProfiles;
use GuzzleHttp\Psr7\Uri;
use Psr\Http\Message\UriInterface;
class CrawlInternalUrls extends CrawlProfile
{
protected mixed $baseUrl;
public function __construct($baseUrl)
{
if (! $baseUrl instanceof UriInterface) {
$baseUrl = new Uri($baseUrl);
}
$this->baseUrl = $baseUrl;
}
public function shouldCrawl(UriInterface $url): bool
{
return $this->baseUrl->getHost() === $url->getHost();
}
}

View File

@@ -0,0 +1,10 @@
<?php
namespace Spatie\Crawler\CrawlProfiles;
use Psr\Http\Message\UriInterface;
abstract class CrawlProfile
{
abstract public function shouldCrawl(UriInterface $url): bool;
}

View File

@@ -0,0 +1,30 @@
<?php
namespace Spatie\Crawler\CrawlProfiles;
use GuzzleHttp\Psr7\Uri;
use Psr\Http\Message\UriInterface;
class CrawlSubdomains extends CrawlProfile
{
protected mixed $baseUrl;
public function __construct($baseUrl)
{
if (! $baseUrl instanceof UriInterface) {
$baseUrl = new Uri($baseUrl);
}
$this->baseUrl = $baseUrl;
}
public function shouldCrawl(UriInterface $url): bool
{
return $this->isSubdomainOfHost($url);
}
public function isSubdomainOfHost(UriInterface $url): bool
{
return str_ends_with($url->getHost(), $this->baseUrl->getHost());
}
}

View File

@@ -0,0 +1,102 @@
<?php
namespace Spatie\Crawler\CrawlQueues;
use Psr\Http\Message\UriInterface;
use Spatie\Crawler\CrawlUrl;
use Spatie\Crawler\Exceptions\InvalidUrl;
use Spatie\Crawler\Exceptions\UrlNotFoundByIndex;
class ArrayCrawlQueue implements CrawlQueue
{
/**
* All known URLs, indexed by URL string.
*
* @var CrawlUrl[]
*/
protected array $urls = [];
/**
* Pending URLs, indexed by URL string.
*
* @var CrawlUrl[]
*/
protected array $pendingUrls = [];
public function add(CrawlUrl $crawlUrl): CrawlQueue
{
$urlString = (string) $crawlUrl->url;
if (! isset($this->urls[$urlString])) {
$crawlUrl->setId($urlString);
$this->urls[$urlString] = $crawlUrl;
$this->pendingUrls[$urlString] = $crawlUrl;
}
return $this;
}
public function hasPendingUrls(): bool
{
return (bool) $this->pendingUrls;
}
public function getUrlById($id): CrawlUrl
{
if (! isset($this->urls[$id])) {
throw new UrlNotFoundByIndex("Crawl url {$id} not found in collection.");
}
return $this->urls[$id];
}
public function hasAlreadyBeenProcessed(CrawlUrl $crawlUrl): bool
{
$urlString = (string) $crawlUrl->url;
if (isset($this->pendingUrls[$urlString])) {
return false;
}
if (isset($this->urls[$urlString])) {
return true;
}
return false;
}
public function markAsProcessed(CrawlUrl $crawlUrl): void
{
$urlString = (string) $crawlUrl->url;
unset($this->pendingUrls[$urlString]);
}
public function getProcessedUrlCount(): int
{
return count($this->urls) - count($this->pendingUrls);
}
public function has(CrawlUrl|UriInterface $crawlUrl): bool
{
if ($crawlUrl instanceof CrawlUrl) {
$urlString = (string) $crawlUrl->url;
} elseif ($crawlUrl instanceof UriInterface) {
$urlString = (string) $crawlUrl;
} else {
throw InvalidUrl::unexpectedType($crawlUrl);
}
return isset($this->urls[$urlString]);
}
public function getPendingUrl(): ?CrawlUrl
{
foreach ($this->pendingUrls as $pendingUrl) {
return $pendingUrl;
}
return null;
}
}

View File

@@ -0,0 +1,25 @@
<?php
namespace Spatie\Crawler\CrawlQueues;
use Psr\Http\Message\UriInterface;
use Spatie\Crawler\CrawlUrl;
interface CrawlQueue
{
public function add(CrawlUrl $url): self;
public function has(CrawlUrl|UriInterface $crawlUrl): bool;
public function hasPendingUrls(): bool;
public function getUrlById($id): CrawlUrl;
public function getPendingUrl(): ?CrawlUrl;
public function hasAlreadyBeenProcessed(CrawlUrl $url): bool;
public function markAsProcessed(CrawlUrl $crawlUrl): void;
public function getProcessedUrlCount(): int;
}

50
vendor/spatie/crawler/src/CrawlUrl.php vendored Executable file
View File

@@ -0,0 +1,50 @@
<?php
namespace Spatie\Crawler;
use Psr\Http\Message\UriInterface;
class CrawlUrl
{
public UriInterface $url;
public ?UriInterface $foundOnUrl = null;
public ?string $linkText = null;
protected mixed $id;
public static function create(
UriInterface $url,
?UriInterface $foundOnUrl = null,
$id = null,
?string $linkText = null,
): static {
$static = new static($url, $foundOnUrl, linkText: $linkText);
if ($id !== null) {
$static->setId($id);
}
return $static;
}
protected function __construct(UriInterface $url, $foundOnUrl = null, $linkText = null)
{
$this->url = $url;
$this->foundOnUrl = $foundOnUrl;
$this->linkText = $linkText;
}
public function getId(): mixed
{
return $this->id;
}
public function setId($id): void
{
$this->id = $id;
}
}

644
vendor/spatie/crawler/src/Crawler.php vendored Executable file
View File

@@ -0,0 +1,644 @@
<?php
namespace Spatie\Crawler;
use Generator;
use GuzzleHttp\Client;
use GuzzleHttp\Pool;
use GuzzleHttp\Psr7\Request;
use GuzzleHttp\Psr7\Uri;
use GuzzleHttp\RequestOptions;
use Psr\Http\Message\UriInterface;
use Spatie\Browsershot\Browsershot;
use Spatie\Crawler\CrawlObservers\CrawlObserver;
use Spatie\Crawler\CrawlObservers\CrawlObserverCollection;
use Spatie\Crawler\CrawlProfiles\CrawlAllUrls;
use Spatie\Crawler\CrawlProfiles\CrawlProfile;
use Spatie\Crawler\CrawlQueues\ArrayCrawlQueue;
use Spatie\Crawler\CrawlQueues\CrawlQueue;
use Spatie\Crawler\Exceptions\InvalidCrawlRequestHandler;
use Spatie\Crawler\Handlers\CrawlRequestFailed;
use Spatie\Crawler\Handlers\CrawlRequestFulfilled;
use Spatie\Crawler\UrlParsers\LinkUrlParser;
use Spatie\Robots\RobotsTxt;
use Tree\Node\Node;
class Crawler
{
public const DEFAULT_USER_AGENT = '*';
protected UriInterface $baseUrl;
protected CrawlObserverCollection $crawlObservers;
protected CrawlProfile $crawlProfile;
protected CrawlQueue $crawlQueue;
protected int $totalUrlCount = 0;
protected int $currentUrlCount = 0;
protected ?int $totalCrawlLimit = null;
protected ?int $currentCrawlLimit = null;
protected ?int $startedAt = null;
protected int $executionTime = 0;
protected ?int $totalExecutionTimeLimit = null;
protected ?int $currentExecutionTimeLimit = null;
protected int $maximumResponseSize = 1024 * 1024 * 2;
protected ?int $maximumDepth = null;
protected bool $respectRobots = true;
protected bool $rejectNofollowLinks = true;
protected Node $depthTree;
protected bool $executeJavaScript = false;
protected ?Browsershot $browsershot = null;
protected ?RobotsTxt $robotsTxt = null;
protected string $crawlRequestFulfilledClass;
protected string $crawlRequestFailedClass;
protected string $urlParserClass;
protected int $delayBetweenRequests = 0;
protected array $allowedMimeTypes = [];
protected string $defaultScheme = 'http';
protected static array $defaultClientOptions = [
RequestOptions::COOKIES => true,
RequestOptions::CONNECT_TIMEOUT => 10,
RequestOptions::TIMEOUT => 10,
RequestOptions::ALLOW_REDIRECTS => false,
RequestOptions::HEADERS => [
'User-Agent' => self::DEFAULT_USER_AGENT,
],
];
public static function create(array $clientOptions = []): static
{
$clientOptions = (count($clientOptions))
? $clientOptions
: static::$defaultClientOptions;
$client = new Client($clientOptions);
return new static($client);
}
public function __construct(
protected Client $client,
protected int $concurrency = 10,
) {
$this->crawlProfile = new CrawlAllUrls;
$this->crawlQueue = new ArrayCrawlQueue;
$this->crawlObservers = new CrawlObserverCollection;
$this->crawlRequestFulfilledClass = CrawlRequestFulfilled::class;
$this->crawlRequestFailedClass = CrawlRequestFailed::class;
$this->urlParserClass = LinkUrlParser::class;
}
public function getDefaultScheme(): string
{
return $this->defaultScheme;
}
public function setDefaultScheme(string $defaultScheme): self
{
$this->defaultScheme = $defaultScheme;
return $this;
}
public function setConcurrency(int $concurrency): self
{
$this->concurrency = $concurrency;
return $this;
}
public function setMaximumResponseSize(int $maximumResponseSizeInBytes): self
{
$this->maximumResponseSize = $maximumResponseSizeInBytes;
return $this;
}
public function getMaximumResponseSize(): ?int
{
return $this->maximumResponseSize;
}
public function setTotalCrawlLimit(int $totalCrawlLimit): self
{
$this->totalCrawlLimit = $totalCrawlLimit;
return $this;
}
public function getTotalCrawlLimit(): ?int
{
return $this->totalCrawlLimit;
}
public function getTotalCrawlCount(): int
{
return $this->totalUrlCount;
}
public function setCurrentCrawlLimit(int $currentCrawlLimit): self
{
$this->currentCrawlLimit = $currentCrawlLimit;
return $this;
}
public function getCurrentCrawlLimit(): ?int
{
return $this->currentCrawlLimit;
}
public function getCurrentCrawlCount(): int
{
return $this->currentUrlCount;
}
public function setTotalExecutionTimeLimit(int $totalExecutionTimeLimitInSecond): self
{
$this->totalExecutionTimeLimit = $totalExecutionTimeLimitInSecond;
return $this;
}
public function getTotalExecutionTimeLimit(): ?int
{
return $this->totalExecutionTimeLimit;
}
public function getTotalExecutionTime(): int
{
return $this->executionTime + $this->getCurrentExecutionTime();
}
public function setCurrentExecutionTimeLimit(int $currentExecutionTimeLimitInSecond): self
{
$this->currentExecutionTimeLimit = $currentExecutionTimeLimitInSecond;
return $this;
}
public function getCurrentExecutionTimeLimit(): ?int
{
return $this->currentExecutionTimeLimit;
}
public function getCurrentExecutionTime(): int
{
if (is_null($this->startedAt)) {
return 0;
}
return time() - $this->startedAt;
}
public function setMaximumDepth(int $maximumDepth): self
{
$this->maximumDepth = $maximumDepth;
return $this;
}
public function getMaximumDepth(): ?int
{
return $this->maximumDepth;
}
public function setDelayBetweenRequests(int $delayInMilliseconds): self
{
$this->delayBetweenRequests = ($delayInMilliseconds * 1000);
return $this;
}
public function getDelayBetweenRequests(): int
{
return $this->delayBetweenRequests;
}
public function setParseableMimeTypes(array $types): self
{
$this->allowedMimeTypes = $types;
return $this;
}
public function getParseableMimeTypes(): array
{
return $this->allowedMimeTypes;
}
public function ignoreRobots(): self
{
$this->respectRobots = false;
return $this;
}
public function respectRobots(): self
{
$this->respectRobots = true;
return $this;
}
public function mustRespectRobots(): bool
{
return $this->respectRobots;
}
public function acceptNofollowLinks(): self
{
$this->rejectNofollowLinks = false;
return $this;
}
public function rejectNofollowLinks(): self
{
$this->rejectNofollowLinks = true;
return $this;
}
public function mustRejectNofollowLinks(): bool
{
return $this->rejectNofollowLinks;
}
public function getRobotsTxt(): ?RobotsTxt
{
return $this->robotsTxt;
}
public function setCrawlQueue(CrawlQueue $crawlQueue): self
{
$this->crawlQueue = $crawlQueue;
return $this;
}
public function getCrawlQueue(): CrawlQueue
{
return $this->crawlQueue;
}
public function executeJavaScript(): self
{
$this->executeJavaScript = true;
return $this;
}
public function doNotExecuteJavaScript(): self
{
$this->executeJavaScript = false;
return $this;
}
public function mayExecuteJavascript(): bool
{
return $this->executeJavaScript;
}
public function setCrawlObserver(CrawlObserver|array $crawlObservers): self
{
if (! is_array($crawlObservers)) {
$crawlObservers = [$crawlObservers];
}
return $this->setCrawlObservers($crawlObservers);
}
public function setCrawlObservers(array $crawlObservers): self
{
$this->crawlObservers = new CrawlObserverCollection($crawlObservers);
return $this;
}
public function addCrawlObserver(CrawlObserver $crawlObserver): self
{
$this->crawlObservers->addObserver($crawlObserver);
return $this;
}
public function getCrawlObservers(): CrawlObserverCollection
{
return $this->crawlObservers;
}
public function setCrawlProfile(CrawlProfile $crawlProfile): self
{
$this->crawlProfile = $crawlProfile;
return $this;
}
public function getCrawlProfile(): CrawlProfile
{
return $this->crawlProfile;
}
public function setCrawlFulfilledHandlerClass(string $crawlRequestFulfilledClass): self
{
$baseClass = CrawlRequestFulfilled::class;
if (! is_subclass_of($crawlRequestFulfilledClass, $baseClass)) {
throw InvalidCrawlRequestHandler::doesNotExtendBaseClass($crawlRequestFulfilledClass, $baseClass);
}
$this->crawlRequestFulfilledClass = $crawlRequestFulfilledClass;
return $this;
}
public function setCrawlFailedHandlerClass(string $crawlRequestFailedClass): self
{
$baseClass = CrawlRequestFailed::class;
if (! is_subclass_of($crawlRequestFailedClass, $baseClass)) {
throw InvalidCrawlRequestHandler::doesNotExtendBaseClass($crawlRequestFailedClass, $baseClass);
}
$this->crawlRequestFailedClass = $crawlRequestFailedClass;
return $this;
}
public function setUrlParserClass(string $urlParserClass): self
{
$this->urlParserClass = $urlParserClass;
return $this;
}
public function getUrlParserClass(): string
{
return $this->urlParserClass;
}
public function setBrowsershot(Browsershot $browsershot)
{
$this->browsershot = $browsershot;
return $this;
}
public function setUserAgent(string $userAgent): self
{
$clientOptions = $this->client->getConfig();
$headers = array_change_key_case($clientOptions['headers']);
$headers['user-agent'] = $userAgent;
$clientOptions['headers'] = $headers;
$this->client = new Client($clientOptions);
return $this;
}
public function getUserAgent(): string
{
$headers = $this->client->getConfig('headers');
foreach (array_keys($headers) as $name) {
if (strtolower($name) === 'user-agent') {
return (string) $headers[$name];
}
}
return static::DEFAULT_USER_AGENT;
}
public function getBrowsershot(): Browsershot
{
if (! $this->browsershot) {
$this->browsershot = new Browsershot;
}
return $this->browsershot;
}
public function getBaseUrl(): UriInterface
{
return $this->baseUrl;
}
public function startCrawling(UriInterface|string $baseUrl)
{
$this->startedAt = time();
if (! $baseUrl instanceof UriInterface) {
$baseUrl = new Uri($baseUrl);
}
if ($baseUrl->getScheme() === '') {
$baseUrl = $baseUrl->withScheme($this->defaultScheme);
}
if ($baseUrl->getPath() === '') {
$baseUrl = $baseUrl->withPath('/');
}
$this->totalUrlCount = $this->crawlQueue->getProcessedUrlCount();
$this->baseUrl = $baseUrl;
$crawlUrl = CrawlUrl::create($this->baseUrl);
if ($this->respectRobots) {
$this->robotsTxt = $this->createRobotsTxt($crawlUrl->url);
}
if ($this->shouldAddToCrawlQueue($crawlUrl)) {
$this->addToCrawlQueue($crawlUrl);
}
$this->depthTree = new Node((string) $this->baseUrl);
$this->startCrawlingQueue();
foreach ($this->crawlObservers as $crawlObserver) {
$crawlObserver->finishedCrawling();
}
$this->executionTime += time() - $this->startedAt;
$this->startedAt = null; // To reset currentExecutionTime
}
public function addToDepthTree(UriInterface $url, UriInterface $parentUrl, ?Node $node = null, ?UriInterface $originalUrl = null): ?Node
{
if (is_null($this->maximumDepth)) {
return new Node((string) $url);
}
$node = $node ?? $this->depthTree;
$returnNode = null;
if ($node->getValue() === (string) $parentUrl || $node->getValue() === (string) $originalUrl) {
$newNode = new Node((string) $url);
$node->addChild($newNode);
return $newNode;
}
foreach ($node->getChildren() as $currentNode) {
$returnNode = $this->addToDepthTree($url, $parentUrl, $currentNode, $originalUrl);
if (! is_null($returnNode)) {
break;
}
}
return $returnNode;
}
protected function shouldAddToCrawlQueue($crawlUrl): bool
{
if (! $this->respectRobots) {
return true;
}
if ($this->robotsTxt === null) {
return false;
}
if ($this->robotsTxt->allows((string) $crawlUrl->url, $this->getUserAgent())) {
return true;
}
return false;
}
protected function startCrawlingQueue(): void
{
while (
$this->reachedCrawlLimits() === false &&
$this->reachedTimeLimits() === false &&
$this->crawlQueue->hasPendingUrls()
) {
$pool = new Pool($this->client, $this->getCrawlRequests(), [
'concurrency' => $this->concurrency,
'options' => $this->client->getConfig(),
'fulfilled' => new $this->crawlRequestFulfilledClass($this),
'rejected' => new $this->crawlRequestFailedClass($this),
]);
$promise = $pool->promise();
$promise->wait();
}
}
protected function createRobotsTxt(UriInterface $uri): RobotsTxt
{
return RobotsTxt::create($uri->withPath('/robots.txt'));
}
protected function getCrawlRequests(): Generator
{
while (
$this->reachedCrawlLimits() === false &&
$this->reachedTimeLimits() === false &&
$crawlUrl = $this->crawlQueue->getPendingUrl()
) {
if (
$this->crawlProfile->shouldCrawl($crawlUrl->url) === false ||
$this->crawlQueue->hasAlreadyBeenProcessed($crawlUrl)
) {
$this->crawlQueue->markAsProcessed($crawlUrl);
continue;
}
foreach ($this->crawlObservers as $crawlObserver) {
$crawlObserver->willCrawl($crawlUrl->url, $crawlUrl->linkText);
}
$this->totalUrlCount++;
$this->currentUrlCount++;
$this->crawlQueue->markAsProcessed($crawlUrl);
yield $crawlUrl->getId() => new Request('GET', $crawlUrl->url);
}
}
public function addToCrawlQueue(CrawlUrl $crawlUrl): self
{
if (! $this->getCrawlProfile()->shouldCrawl($crawlUrl->url)) {
return $this;
}
if ($this->getCrawlQueue()->has($crawlUrl->url)) {
return $this;
}
$this->crawlQueue->add($crawlUrl);
return $this;
}
public function reachedCrawlLimits(): bool
{
$totalCrawlLimit = $this->getTotalCrawlLimit();
if (! is_null($totalCrawlLimit) && $this->getTotalCrawlCount() >= $totalCrawlLimit) {
return true;
}
$currentCrawlLimit = $this->getCurrentCrawlLimit();
if (! is_null($currentCrawlLimit) && $this->getCurrentCrawlCount() >= $currentCrawlLimit) {
return true;
}
return false;
}
public function reachedTimeLimits(): bool
{
$totalExecutionTimeLimit = $this->getTotalExecutionTimeLimit();
if (! is_null($totalExecutionTimeLimit) && $this->getTotalExecutionTime() >= $totalExecutionTimeLimit) {
return true;
}
$currentExecutionTimeLimit = $this->getCurrentExecutionTimeLimit();
if (! is_null($currentExecutionTimeLimit) && $this->getCurrentExecutionTime() >= $currentExecutionTimeLimit) {
return true;
}
return false;
}
}

58
vendor/spatie/crawler/src/CrawlerRobots.php vendored Executable file
View File

@@ -0,0 +1,58 @@
<?php
namespace Spatie\Crawler;
use Spatie\Robots\RobotsHeaders;
use Spatie\Robots\RobotsMeta;
class CrawlerRobots
{
protected RobotsHeaders $robotsHeaders;
protected RobotsMeta $robotsMeta;
protected bool $mustRespectRobots;
public function __construct(array $headers, string $body, bool $mustRespectRobots)
{
$this->robotsHeaders = RobotsHeaders::create($headers);
$this->robotsMeta = RobotsMeta::create($body);
$this->mustRespectRobots = $mustRespectRobots;
}
public function mayIndex(): bool
{
if (! $this->mustRespectRobots) {
return true;
}
if (! $this->robotsHeaders->mayIndex()) {
return false;
}
if (! $this->robotsMeta->mayIndex()) {
return false;
}
return true;
}
public function mayFollow(): bool
{
if (! $this->mustRespectRobots) {
return true;
}
if (! $this->robotsHeaders->mayFollow()) {
return false;
}
if (! $this->robotsMeta->mayFollow()) {
return false;
}
return true;
}
}

View File

@@ -0,0 +1,13 @@
<?php
namespace Spatie\Crawler\Exceptions;
use RuntimeException;
class InvalidCrawlRequestHandler extends RuntimeException
{
public static function doesNotExtendBaseClass(string $handlerClass, string $baseClass): static
{
return new static("`{$handlerClass} is not a valid handler class. A valid handler class should extend `{$baseClass}`.");
}
}

View File

@@ -0,0 +1,19 @@
<?php
namespace Spatie\Crawler\Exceptions;
use Exception;
use Psr\Http\Message\UriInterface;
use Spatie\Crawler\CrawlUrl;
class InvalidUrl extends Exception
{
public static function unexpectedType(mixed $url): static
{
$crawlUrlClass = CrawlUrl::class;
$uriInterfaceClass = UriInterface::class;
$givenUrlClass = is_object($url) ? get_class($url) : gettype($url);
return new static("You passed an invalid url of type `{$givenUrlClass}`. This should be either a {$crawlUrlClass} or `{$uriInterfaceClass}`");
}
}

View File

@@ -0,0 +1,7 @@
<?php
namespace Spatie\Crawler\Exceptions;
use RuntimeException;
class UrlNotFoundByIndex extends RuntimeException {}

View File

@@ -0,0 +1,31 @@
<?php
namespace Spatie\Crawler\Handlers;
use Exception;
use GuzzleHttp\Exception\ConnectException;
use GuzzleHttp\Exception\RequestException;
use Spatie\Crawler\Crawler;
class CrawlRequestFailed
{
public function __construct(protected Crawler $crawler)
{
//
}
public function __invoke(Exception $exception, $index)
{
if ($exception instanceof ConnectException) {
$exception = new RequestException($exception->getMessage(), $exception->getRequest());
}
if ($exception instanceof RequestException) {
$crawlUrl = $this->crawler->getCrawlQueue()->getUrlById($index);
$this->crawler->getCrawlObservers()->crawlFailed($crawlUrl, $exception);
}
usleep($this->crawler->getDelayBetweenRequests());
}
}

View File

@@ -0,0 +1,173 @@
<?php
namespace Spatie\Crawler\Handlers;
use Exception;
use GuzzleHttp\Exception\RequestException;
use GuzzleHttp\Psr7\Request;
use GuzzleHttp\Psr7\Uri;
use GuzzleHttp\Psr7\Utils;
use GuzzleHttp\RedirectMiddleware;
use Psr\Http\Message\ResponseInterface;
use Psr\Http\Message\StreamInterface;
use Psr\Http\Message\UriInterface;
use Spatie\Crawler\Crawler;
use Spatie\Crawler\CrawlerRobots;
use Spatie\Crawler\CrawlProfiles\CrawlSubdomains;
use Spatie\Crawler\CrawlUrl;
use Spatie\Crawler\ResponseWithCachedBody;
use Spatie\Crawler\UrlParsers\UrlParser;
use Symfony\Component\Process\Exception\ProcessFailedException;
class CrawlRequestFulfilled
{
protected UrlParser $urlParser;
public function __construct(protected Crawler $crawler)
{
$urlParserClass = $this->crawler->getUrlParserClass();
$this->urlParser = new $urlParserClass($this->crawler);
}
public function __invoke(ResponseInterface $response, $index)
{
$body = $this->getBody($response);
if (empty($body)) {
usleep($this->crawler->getDelayBetweenRequests());
return;
}
$robots = new CrawlerRobots(
$response->getHeaders(),
$body,
$this->crawler->mustRespectRobots()
);
$crawlUrl = $this->crawler->getCrawlQueue()->getUrlById($index);
if ($this->crawler->mayExecuteJavaScript()) {
try {
$body = $this->getBodyAfterExecutingJavaScript($crawlUrl->url);
} catch (ProcessFailedException $exception) {
$request = new Request('GET', $crawlUrl->url);
$exception = new RequestException($exception->getMessage(), $request);
$crawlUrl = $this->crawler->getCrawlQueue()->getUrlById($index);
$this->crawler->getCrawlObservers()->crawlFailed($crawlUrl, $exception);
usleep($this->crawler->getDelayBetweenRequests());
return;
}
$response = $response->withBody(Utils::streamFor($body));
}
$responseWithCachedBody = ResponseWithCachedBody::fromGuzzlePsr7Response($response);
$responseWithCachedBody->setCachedBody($body);
if ($robots->mayIndex()) {
$this->handleCrawled($responseWithCachedBody, $crawlUrl);
}
if (! $this->crawler->getCrawlProfile() instanceof CrawlSubdomains) {
if ($crawlUrl->url->getHost() !== $this->crawler->getBaseUrl()->getHost()) {
return;
}
}
if (! $robots->mayFollow()) {
return;
}
$baseUrl = $this->getBaseUrl($response, $crawlUrl);
$originalUrl = $crawlUrl->url;
$this->urlParser->addFromHtml($body, $baseUrl, $originalUrl);
usleep($this->crawler->getDelayBetweenRequests());
}
protected function getBaseUrl(ResponseInterface $response, CrawlUrl $crawlUrl): UriInterface
{
$redirectHistory = $response->getHeader(RedirectMiddleware::HISTORY_HEADER);
if (empty($redirectHistory)) {
return $crawlUrl->url;
}
return new Uri(end($redirectHistory));
}
protected function handleCrawled(ResponseInterface $response, CrawlUrl $crawlUrl): void
{
$this->crawler->getCrawlObservers()->crawled($crawlUrl, $response);
}
protected function getBody(ResponseInterface $response): string
{
$contentType = $response->getHeaderLine('Content-Type');
if (! $this->isMimetypeAllowedToParse($contentType)) {
return '';
}
return $this->convertBodyToString($response->getBody(), $this->crawler->getMaximumResponseSize());
}
protected function convertBodyToString(StreamInterface $bodyStream, $readMaximumBytes = 1024 * 1024 * 2): string
{
if ($bodyStream->isSeekable()) {
$bodyStream->rewind();
}
$body = '';
$chunksToRead = $readMaximumBytes < 512 ? $readMaximumBytes : 512;
for ($bytesRead = 0; $bytesRead < $readMaximumBytes; $bytesRead += $chunksToRead) {
try {
$newDataRead = $bodyStream->read($chunksToRead);
} catch (Exception $exception) {
$newDataRead = null;
}
if (! $newDataRead) {
break;
}
$body .= $newDataRead;
}
return $body;
}
protected function getBodyAfterExecutingJavaScript(UriInterface $url): string
{
$browsershot = $this->crawler->getBrowsershot();
$html = $browsershot->setUrl((string) $url)->bodyHtml();
return html_entity_decode($html);
}
protected function isMimetypeAllowedToParse($contentType): bool
{
if (empty($contentType)) {
return true;
}
if (! count($this->crawler->getParseableMimeTypes())) {
return true;
}
foreach ($this->crawler->getParseableMimeTypes() as $allowedType) {
if (stristr($contentType, $allowedType)) {
return true;
}
}
return false;
}
}

View File

@@ -0,0 +1,32 @@
<?php
namespace Spatie\Crawler;
use GuzzleHttp\Psr7\Response;
use Psr\Http\Message\ResponseInterface;
class ResponseWithCachedBody extends Response
{
protected ?string $cachedBody = null;
public static function fromGuzzlePsr7Response(ResponseInterface $response): static
{
return new static(
$response->getStatusCode(),
$response->getHeaders(),
$response->getBody(),
$response->getProtocolVersion(),
$response->getReasonPhrase()
);
}
public function setCachedBody(?string $body = null): void
{
$this->cachedBody = $body;
}
public function getCachedBody(): ?string
{
return $this->cachedBody;
}
}

20
vendor/spatie/crawler/src/Url.php vendored Executable file
View File

@@ -0,0 +1,20 @@
<?php
namespace Spatie\Crawler;
use GuzzleHttp\Psr7\Uri;
class Url extends Uri
{
public function __construct(
protected string $link,
protected ?string $linkText,
) {
parent::__construct($link);
}
public function linkText(): ?string
{
return $this->linkText;
}
}

View File

@@ -0,0 +1,125 @@
<?php
namespace Spatie\Crawler\UrlParsers;
use Illuminate\Support\Collection;
use InvalidArgumentException;
use Psr\Http\Message\UriInterface;
use Spatie\Crawler\Crawler;
use Spatie\Crawler\CrawlUrl;
use Spatie\Crawler\Url;
use Symfony\Component\DomCrawler\Crawler as DomCrawler;
use Symfony\Component\DomCrawler\Link;
use Tree\Node\Node;
class LinkUrlParser implements UrlParser
{
protected Crawler $crawler;
public function __construct(Crawler $crawler)
{
$this->crawler = $crawler;
}
public function addFromHtml(string $html, UriInterface $foundOnUrl, ?UriInterface $originalUrl = null): void
{
$allLinks = $this->extractLinksFromHtml($html, $foundOnUrl);
collect($allLinks)
->filter(fn (Url $url) => $this->hasCrawlableScheme($url))
->map(fn (Url $url) => $this->normalizeUrl($url))
->filter(function (Url $url) use ($foundOnUrl, $originalUrl) {
if (! $node = $this->crawler->addToDepthTree($url, $foundOnUrl, null, $originalUrl)) {
return false;
}
return $this->shouldCrawl($node);
})
->filter(fn (Url $url) => ! str_contains($url->getPath(), '/tel:'))
->each(function (Url $url) use ($foundOnUrl) {
$crawlUrl = CrawlUrl::create($url, $foundOnUrl, linkText: $url->linkText());
$this->crawler->addToCrawlQueue($crawlUrl);
});
}
protected function extractLinksFromHtml(string $html, UriInterface $foundOnUrl): ?Collection
{
$domCrawler = new DomCrawler($html, $foundOnUrl);
return collect($domCrawler->filterXpath('//a | //link[@rel="next" or @rel="prev"]')->links())
->reject(function (Link $link) {
if ($this->isInvalidHrefNode($link)) {
return true;
}
if ($this->crawler->mustRejectNofollowLinks() && str_contains($link->getNode()->getAttribute('rel'), 'nofollow')) {
return true;
}
return false;
})
->map(function (Link $link) {
try {
$linkText = $link->getNode()->textContent;
if ($linkText) {
$linkText = substr($linkText, 0, 4000);
}
return new Url($link->getUri(), $linkText);
} catch (InvalidArgumentException $exception) {
return null;
}
})
->filter();
}
protected function hasCrawlableScheme(UriInterface $uri): bool
{
return in_array($uri->getScheme(), ['http', 'https']);
}
protected function normalizeUrl(UriInterface $url): UriInterface
{
return $url->withFragment('');
}
protected function shouldCrawl(Node $node): bool
{
$mustRespectRobots = $this->crawler->mustRespectRobots();
$robotsTxt = $this->crawler->getRobotsTxt();
if ($mustRespectRobots && $robotsTxt !== null) {
$isAllowed = $robotsTxt->allows($node->getValue(), $this->crawler->getUserAgent());
if (! $isAllowed) {
return false;
}
}
$maximumDepth = $this->crawler->getMaximumDepth();
if (is_null($maximumDepth)) {
return true;
}
return $node->getDepth() <= $maximumDepth;
}
protected function isInvalidHrefNode(Link $link): bool
{
if ($link->getNode()->nodeName !== 'a') {
return false;
}
if ($link->getNode()->nextSibling !== null) {
return false;
}
if ($link->getNode()->childNodes->length !== 0) {
return false;
}
return true;
}
}

View File

@@ -0,0 +1,95 @@
<?php
namespace Spatie\Crawler\UrlParsers;
use Illuminate\Support\Collection;
use InvalidArgumentException;
use Psr\Http\Message\UriInterface;
use Spatie\Crawler\Crawler;
use Spatie\Crawler\CrawlUrl;
use Spatie\Crawler\Url;
use Symfony\Component\DomCrawler\Crawler as DomCrawler;
use Tree\Node\Node;
class SitemapUrlParser implements UrlParser
{
protected Crawler $crawler;
public function __construct(Crawler $crawler)
{
$this->crawler = $crawler;
}
public function addFromHtml(string $html, UriInterface $foundOnUrl, ?UriInterface $originalUrl = null): void
{
$allLinks = $this->extractLinksFromHtml($html, $foundOnUrl);
collect($allLinks)
->filter(fn (Url $url) => $this->hasCrawlableScheme($url))
->map(fn (Url $url) => $this->normalizeUrl($url))
->filter(function (Url $url) use ($foundOnUrl, $originalUrl) {
if (! $node = $this->crawler->addToDepthTree($url, $foundOnUrl, null, $originalUrl)) {
return false;
}
return $this->shouldCrawl($node);
})
->filter(fn (Url $url) => ! str_contains($url->getPath(), '/tel:'))
->each(function (Url $url) use ($foundOnUrl) {
$crawlUrl = CrawlUrl::create($url, $foundOnUrl, linkText: $url->linkText());
$this->crawler->addToCrawlQueue($crawlUrl);
});
}
protected function extractLinksFromHtml(string $html, UriInterface $foundOnUrl): ?Collection
{
$domCrawler = new DomCrawler($html, $foundOnUrl);
return collect($domCrawler->filterXPath('//loc')
->each(function (DomCrawler $node) {
try {
$linkText = $node->text();
if ($linkText) {
$linkText = substr($linkText, 0, 4000);
}
return new Url($linkText, $linkText);
} catch (InvalidArgumentException $exception) {
return null;
}
}));
}
protected function hasCrawlableScheme(UriInterface $uri): bool
{
return in_array($uri->getScheme(), ['http', 'https']);
}
protected function normalizeUrl(UriInterface $url): UriInterface
{
return $url->withFragment('');
}
protected function shouldCrawl(Node $node): bool
{
$mustRespectRobots = $this->crawler->mustRespectRobots();
$robotsTxt = $this->crawler->getRobotsTxt();
if ($mustRespectRobots && $robotsTxt !== null) {
$isAllowed = $robotsTxt->allows($node->getValue(), $this->crawler->getUserAgent());
if (! $isAllowed) {
return false;
}
}
$maximumDepth = $this->crawler->getMaximumDepth();
if (is_null($maximumDepth)) {
return true;
}
return $node->getDepth() <= $maximumDepth;
}
}

View File

@@ -0,0 +1,13 @@
<?php
namespace Spatie\Crawler\UrlParsers;
use Psr\Http\Message\UriInterface;
use Spatie\Crawler\Crawler;
interface UrlParser
{
public function __construct(Crawler $crawler);
public function addFromHtml(string $html, UriInterface $foundOnUrl, ?UriInterface $originalUrl = null): void;
}

35
vendor/spatie/error-solutions/.php_cs.php vendored Executable file
View File

@@ -0,0 +1,35 @@
<?php
$finder = Symfony\Component\Finder\Finder::create()
->in([
__DIR__ . '/src',
__DIR__ . '/tests',
])
->name('*.php')
->notName('*.blade.php')
->ignoreDotFiles(true)
->ignoreVCS(true);
return (new PhpCsFixer\Config())
->setRules([
'@PSR2' => true,
'array_syntax' => ['syntax' => 'short'],
'ordered_imports' => ['sort_algorithm' => 'alpha'],
'no_unused_imports' => true,
'not_operator_with_successor_space' => true,
'trailing_comma_in_multiline' => true,
'phpdoc_scalar' => true,
'unary_operator_spaces' => true,
'binary_operator_spaces' => true,
'blank_line_before_statement' => [
'statements' => ['break', 'continue', 'declare', 'return', 'throw', 'try'],
],
'phpdoc_single_line_var_spacing' => true,
'phpdoc_var_without_name' => true,
'method_argument_space' => [
'on_multiline' => 'ensure_fully_multiline',
'keep_multiple_spaces_after_comma' => true,
],
'single_trait_insert_per_statement' => true,
])
->setFinder($finder);

90
vendor/spatie/error-solutions/CHANGELOG.md vendored Executable file
View File

@@ -0,0 +1,90 @@
# Changelog
All notable changes to `error-solutions` will be documented in this file.
## 1.1.2 - 2024-12-11
### What's Changed
* Documentation link should follow the latest major version by @mshukurlu in https://github.com/spatie/error-solutions/pull/17
* Replace implicitly nullable parameters for PHP 8.4 by @txdFabio in https://github.com/spatie/error-solutions/pull/22
### New Contributors
* @mshukurlu made their first contribution in https://github.com/spatie/error-solutions/pull/17
* @txdFabio made their first contribution in https://github.com/spatie/error-solutions/pull/22
**Full Changelog**: https://github.com/spatie/error-solutions/compare/1.1.1...1.1.2
## 1.1.1 - 2024-07-25
### What's Changed
* Fix OpenAI response text links by @Lukaaashek in https://github.com/spatie/error-solutions/pull/9
### New Contributors
* @Lukaaashek made their first contribution in https://github.com/spatie/error-solutions/pull/9
**Full Changelog**: https://github.com/spatie/error-solutions/compare/1.1.0...1.1.1
## 1.1.0 - 2024-07-22
### What's Changed
* Allow to customize OpenAI Model by @arnebr in https://github.com/spatie/error-solutions/pull/7
### New Contributors
* @arnebr made their first contribution in https://github.com/spatie/error-solutions/pull/7
**Full Changelog**: https://github.com/spatie/error-solutions/compare/1.0.5...1.1.0
## 1.0.5 - 2024-07-09
### What's Changed
* Legacy `RunnableSolution` should continue to extend legacy `Solution` by @duncanmcclean in https://github.com/spatie/error-solutions/pull/6
### New Contributors
* @duncanmcclean made their first contribution in https://github.com/spatie/error-solutions/pull/6
**Full Changelog**: https://github.com/spatie/error-solutions/compare/1.0.4...1.0.5
## 1.0.4 - 2024-06-28
**Full Changelog**: https://github.com/spatie/error-solutions/compare/1.0.3...1.0.4
## 1.0.3 - 2024-06-27
**Full Changelog**: https://github.com/spatie/error-solutions/compare/1.0.2...1.0.3
## 1.0.2 - 2024-06-26
### What's Changed
* Fix AI solutions
* Bump dependabot/fetch-metadata from 1.6.0 to 2.1.0 by @dependabot in https://github.com/spatie/error-solutions/pull/1
### New Contributors
* @dependabot made their first contribution in https://github.com/spatie/error-solutions/pull/1
**Full Changelog**: https://github.com/spatie/error-solutions/compare/0.0.1...1.0.2
## 1.0.1 - 2024-06-21
- Add the legacy string comperator
**Full Changelog**: https://github.com/spatie/error-solutions/compare/1.0.0...1.0.1
## 1.0.0 - 2024-06-12
- Initial release
**Full Changelog**: https://github.com/spatie/error-solutions/compare/0.0.1...1.0.0
## 0.0.1 - 2024-06-11
**Full Changelog**: https://github.com/spatie/error-solutions/commits/0.0.1

21
vendor/spatie/error-solutions/LICENSE.md vendored Executable file
View File

@@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) Spatie <ruben@spatie.be>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

57
vendor/spatie/error-solutions/README.md vendored Executable file
View File

@@ -0,0 +1,57 @@
# Error solutions
[![Latest Version on Packagist](https://img.shields.io/packagist/v/spatie/error-solutions.svg?style=flat-square)](https://packagist.org/packages/spatie/error-solutions)
[![Tests](https://img.shields.io/github/actions/workflow/status/spatie/error-solutions/run-tests.yml?branch=main&label=tests&style=flat-square)](https://github.com/spatie/error-solutions/actions/workflows/run-tests.yml)
[![Total Downloads](https://img.shields.io/packagist/dt/spatie/error-solutions.svg?style=flat-square)](https://packagist.org/packages/spatie/error-solutions)
At Spatie we develop multiple packages handling errors and providing solutions for these errors. This package is a collection of all these solutions.
## Support us
[<img src="https://github-ads.s3.eu-central-1.amazonaws.com/error-solutions.jpg?t=1" width="419px" />](https://spatie.be/github-ad-click/error-solutions)
We invest a lot of resources into creating [best in class open source packages](https://spatie.be/open-source). You can support us by [buying one of our paid products](https://spatie.be/open-source/support-us).
We highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using. You'll find our address on [our contact page](https://spatie.be/about-us). We publish all received postcards on [our virtual postcard wall](https://spatie.be/open-source/postcards).
## Installation
You can install the package via composer:
```bash
composer require spatie/error-solutions
```
## Usage
We've got some excellent documentation on how to use solutions:
- [Flare](https://flareapp.io/docs/ignition/solutions/implementing-solutions)
- [Ignition](https://github.com/spatie/ignition/?tab=readme-ov-file#displaying-solutions)
## Testing
```bash
composer test
```
## Changelog
Please see [CHANGELOG](CHANGELOG.md) for more information on what has changed recently.
## Contributing
Please see [CONTRIBUTING](https://github.com/spatie/.github/blob/main/CONTRIBUTING.md) for details.
## Security Vulnerabilities
Please review [our security policy](../../security/policy) on how to report security vulnerabilities.
## Credits
- [Ruben Van Assche](https://github.com/rubenvanassche)
- [All Contributors](../../contributors)
## License
The MIT License (MIT). Please see [License File](LICENSE.md) for more information.

69
vendor/spatie/error-solutions/composer.json vendored Executable file
View File

@@ -0,0 +1,69 @@
{
"name" : "spatie/error-solutions",
"description" : "This is my package error-solutions",
"keywords" : [
"Spatie",
"error-solutions"
],
"homepage" : "https://github.com/spatie/error-solutions",
"license" : "MIT",
"authors" : [
{
"name" : "Ruben Van Assche",
"email" : "ruben@spatie.be",
"role" : "Developer"
}
],
"require" : {
"php": "^8.0"
},
"require-dev" : {
"livewire/livewire": "^2.11|^3.5.20",
"illuminate/support": "^10.0|^11.0|^12.0",
"illuminate/broadcasting" : "^10.0|^11.0|^12.0",
"openai-php/client": "^0.10.1",
"illuminate/cache" : "^10.0|^11.0|^12.0",
"pestphp/pest" : "^2.20|^3.0",
"phpstan/phpstan" : "^2.1",
"psr/simple-cache-implementation" : "^3.0",
"psr/simple-cache" : "^3.0",
"spatie/ray" : "^1.28",
"symfony/cache" : "^5.4|^6.0|^7.0",
"symfony/process" : "^5.4|^6.0|^7.0",
"vlucas/phpdotenv" : "^5.5",
"orchestra/testbench": "8.22.3|^9.0|^10.0"
},
"autoload" : {
"psr-4" : {
"Spatie\\ErrorSolutions\\" : "src",
"Spatie\\Ignition\\" : "legacy/ignition",
"Spatie\\LaravelIgnition\\" : "legacy/laravel-ignition"
}
},
"autoload-dev" : {
"psr-4" : {
"Spatie\\ErrorSolutions\\Tests\\" : "tests"
}
},
"suggest" : {
"openai-php/client" : "Require get solutions from OpenAI",
"simple-cache-implementation" : "To cache solutions from OpenAI"
},
"scripts" : {
"analyse" : "vendor/bin/phpstan analyse",
"baseline" : "vendor/bin/phpstan analyse --generate-baseline",
"test" : "vendor/bin/pest",
"test-coverage" : "vendor/bin/pest --coverage",
"format" : "vendor/bin/pint"
},
"config" : {
"sort-packages" : true,
"allow-plugins" : {
"pestphp/pest-plugin": true,
"phpstan/extension-installer": true,
"php-http/discovery": false
}
},
"minimum-stability" : "dev",
"prefer-stable" : true
}

View File

@@ -0,0 +1,7 @@
<?php
namespace Spatie\Ignition\Contracts;
class BaseSolution extends \Spatie\ErrorSolutions\Contracts\BaseSolution implements Solution
{
}

View File

@@ -0,0 +1,7 @@
<?php
namespace Spatie\Ignition\Contracts;
interface HasSolutionsForThrowable extends \Spatie\ErrorSolutions\Contracts\HasSolutionsForThrowable
{
}

View File

@@ -0,0 +1,7 @@
<?php
namespace Spatie\Ignition\Contracts;
interface ProvidesSolution extends \Spatie\ErrorSolutions\Contracts\ProvidesSolution
{
}

View File

@@ -0,0 +1,16 @@
<?php
namespace Spatie\Ignition\Contracts;
interface RunnableSolution extends Solution
{
public function getSolutionActionDescription(): string;
public function getRunButtonText(): string;
/** @param array<string, mixed> $parameters */
public function run(array $parameters = []): void;
/** @return array<string, mixed> */
public function getRunParameters(): array;
}

View File

@@ -0,0 +1,7 @@
<?php
namespace Spatie\Ignition\Contracts;
interface Solution extends \Spatie\ErrorSolutions\Contracts\Solution
{
}

View File

@@ -0,0 +1,8 @@
<?php
namespace Spatie\Ignition\Contracts;
interface SolutionProviderRepository extends \Spatie\ErrorSolutions\Contracts\SolutionProviderRepository
{
}

View File

@@ -0,0 +1,8 @@
<?php
namespace Spatie\Ignition\Solutions\OpenAi;
class DummyCache extends \Spatie\ErrorSolutions\Solutions\OpenAi\DummyCache
{
}

View File

@@ -0,0 +1,7 @@
<?php
namespace Spatie\Ignition\Solutions\OpenAi;
class OpenAiPromptViewModel extends \Spatie\ErrorSolutions\Solutions\OpenAi\OpenAiPromptViewModel
{
}

View File

@@ -0,0 +1,10 @@
<?php
namespace Spatie\Ignition\Solutions\OpenAi;
use Spatie\Ignition\Contracts\Solution;
class OpenAiSolution extends \Spatie\ErrorSolutions\Solutions\OpenAi\OpenAiSolution implements Solution
{
}

View File

@@ -0,0 +1,10 @@
<?php
namespace Spatie\Ignition\Solutions\OpenAi;
use Spatie\Ignition\Contracts\HasSolutionsForThrowable;
class OpenAiSolutionProvider extends \Spatie\ErrorSolutions\Solutions\OpenAi\OpenAiSolutionProvider implements HasSolutionsForThrowable
{
}

View File

@@ -0,0 +1,8 @@
<?php
namespace Spatie\Ignition\Solutions\OpenAi;
class OpenAiSolutionResponse extends \Spatie\ErrorSolutions\Solutions\OpenAi\OpenAiSolutionResponse
{
}

View File

@@ -0,0 +1,10 @@
<?php
namespace Spatie\Ignition\Solutions\SolutionProviders;
use Spatie\ErrorSolutions\SolutionProviders\BadMethodCallSolutionProvider as BaseBadMethodCallSolutionProviderAlias;
use Spatie\Ignition\Contracts\HasSolutionsForThrowable;
class BadMethodCallSolutionProvider extends BaseBadMethodCallSolutionProviderAlias implements HasSolutionsForThrowable
{
}

View File

@@ -0,0 +1,10 @@
<?php
namespace Spatie\Ignition\Solutions\SolutionProviders;
use Spatie\ErrorSolutions\SolutionProviders\MergeConflictSolutionProvider as BaseMergeConflictSolutionProviderAlias;
use Spatie\Ignition\Contracts\HasSolutionsForThrowable;
class MergeConflictSolutionProvider extends BaseMergeConflictSolutionProviderAlias implements HasSolutionsForThrowable
{
}

View File

@@ -0,0 +1,10 @@
<?php
namespace Spatie\Ignition\Solutions\SolutionProviders;
use Spatie\ErrorSolutions\SolutionProviderRepository as BaseSolutionProviderRepositoryAlias;
use Spatie\Ignition\Contracts\SolutionProviderRepository as SolutionProviderRepositoryContract;
class SolutionProviderRepository extends BaseSolutionProviderRepositoryAlias implements SolutionProviderRepositoryContract
{
}

View File

@@ -0,0 +1,11 @@
<?php
namespace Spatie\Ignition\Solutions\SolutionProviders;
use Spatie\ErrorSolutions\SolutionProviders\UndefinedPropertySolutionProvider as BaseUndefinedPropertySolutionProviderAlias;
use Spatie\Ignition\Contracts\HasSolutionsForThrowable;
class UndefinedPropertySolutionProvider extends BaseUndefinedPropertySolutionProviderAlias implements HasSolutionsForThrowable
{
}

View File

@@ -0,0 +1,7 @@
<?php
namespace Spatie\Ignition\Solutions;
class SolutionTransformer extends \Spatie\ErrorSolutions\Solutions\SolutionTransformer
{
}

View File

@@ -0,0 +1,10 @@
<?php
namespace Spatie\Ignition\Solutions;
use Spatie\ErrorSolutions\Solutions\SuggestCorrectVariableNameSolution as BaseSuggestCorrectVariableNameSolutionAlias;
use Spatie\Ignition\Contracts\Solution;
class SuggestCorrectVariableNameSolution extends BaseSuggestCorrectVariableNameSolutionAlias implements Solution
{
}

View File

@@ -0,0 +1,11 @@
<?php
namespace Spatie\Ignition\Solutions;
use Spatie\ErrorSolutions\Solutions\SuggestImportSolution as BaseSuggestImportSolutionAlias;
use Spatie\Ignition\Contracts\Solution;
class SuggestImportSolution extends BaseSuggestImportSolutionAlias implements Solution
{
}

View File

@@ -0,0 +1,11 @@
<?php
namespace Spatie\LaravelIgnition\Solutions;
use Spatie\ErrorSolutions\Solutions\Laravel\GenerateAppKeySolution as BaseGenerateAppKeySolutionAlias;
use Spatie\Ignition\Contracts\Solution;
class GenerateAppKeySolution extends BaseGenerateAppKeySolutionAlias implements Solution
{
}

View File

@@ -0,0 +1,11 @@
<?php
namespace Spatie\LaravelIgnition\Solutions;
use Spatie\ErrorSolutions\Solutions\Laravel\LivewireDiscoverSolution as BaseLivewireDiscoverSolutionAlias;
use Spatie\Ignition\Contracts\Solution;
class LivewireDiscoverSolution extends BaseLivewireDiscoverSolutionAlias implements Solution
{
}

View File

@@ -0,0 +1,11 @@
<?php
namespace Spatie\LaravelIgnition\Solutions;
use Spatie\ErrorSolutions\Solutions\Laravel\MakeViewVariableOptionalSolution as BaseMakeViewVariableOptionalSolutionAlias;
use Spatie\Ignition\Contracts\Solution;
class MakeViewVariableOptionalSolution extends BaseMakeViewVariableOptionalSolutionAlias implements Solution
{
}

View File

@@ -0,0 +1,11 @@
<?php
namespace Spatie\LaravelIgnition\Solutions;
use Spatie\ErrorSolutions\Solutions\Laravel\RunMigrationsSolution as BaseRunMigrationsSolutionAlias;
use Spatie\Ignition\Contracts\Solution;
class RunMigrationsSolution extends BaseRunMigrationsSolutionAlias implements Solution
{
}

View File

@@ -0,0 +1,11 @@
<?php
namespace Spatie\LaravelIgnition\Solutions\SolutionProviders;
use Spatie\ErrorSolutions\SolutionProviders\Laravel\DefaultDbNameSolutionProvider as BaseDefaultDbNameSolutionProviderAlias;
use Spatie\Ignition\Contracts\HasSolutionsForThrowable;
class DefaultDbNameSolutionProvider extends BaseDefaultDbNameSolutionProviderAlias implements HasSolutionsForThrowable
{
}

View File

@@ -0,0 +1,11 @@
<?php
namespace Spatie\LaravelIgnition\Solutions\SolutionProviders;
use Spatie\ErrorSolutions\SolutionProviders\Laravel\GenericLaravelExceptionSolutionProvider as BaseGenericLaravelExceptionSolutionProviderAlias;
use Spatie\Ignition\Contracts\HasSolutionsForThrowable;
class GenericLaravelExceptionSolutionProvider extends BaseGenericLaravelExceptionSolutionProviderAlias implements HasSolutionsForThrowable
{
}

View File

@@ -0,0 +1,11 @@
<?php
namespace Spatie\LaravelIgnition\Solutions\SolutionProviders;
use Spatie\ErrorSolutions\SolutionProviders\Laravel\IncorrectValetDbCredentialsSolutionProvider as BaseIncorrectValetDbCredentialsSolutionProviderAlias;
use Spatie\Ignition\Contracts\HasSolutionsForThrowable;
class IncorrectValetDbCredentialsSolutionProvider extends BaseIncorrectValetDbCredentialsSolutionProviderAlias implements HasSolutionsForThrowable
{
}

Some files were not shown because too many files have changed in this diff Show More